PETSC SparsMatrix initialization error

[Zachary 42!]

Hi everyone,

I am trying to debug this strange behavior.  I am trying to build a PETSC sparse parallel matrix using 4 processors.  This gives me 32 local number of rows (so 128 global number of rows).  But when I pass the local_num_of_rows variable into the reinit function, this is the PETSC error I get:

PETSC ERROR: Nonconforming object sizes
PETSC ERROR: Sum of local lengths 512 does not equal global length 128, my local length 128

Here is my reinit function and necessary vectors:

std::vector<size_type> local_rows_per_process(num_procs,local_num_rows);
std::vector<size_type> local_columns_per_process(num_procs,number_of_columns);

spm.reinit(MPI_COMM_WORLD, dsp, local_rows_per_process, local_columns_per_process, my_proc);

The number of local rows for this example is local_num_rows=32, I printed to check.  Though when it is passed into the reinit function, it looks like it is using the global number of rows.  

I get the same error from the constructor that doesn’t use the dynamic sparsity pattern:

spm.reinit(MPI_COMM_WORLD,global_row_size,global_column_size,local_row_size,local_column_size,number_of_non_zeros);

 

Just for clarifying this constructor, what is “local_rows” and “local_columns” for this constructor?  The documentation just says see the class documentation.  I see where the 4th constructor uses “local_rows_per_process” and this means how many rows do all other processors own and same for the columns so I thought I had that figured out for my constructor with the dynamic sparsity pattern but maybe not.  For this constructor, I just used the local number of rows and columns. 

Can someone please show me what they would do to debug this situation?

Thank you,

Zachary

[Wolfgang Bangerth]

Zachary,

> I am trying to debug this strange behavior.  I am trying to build a PETSC
> sparse parallel matrix using 4 processors.  This gives me 32 local number of
> rows (so 128 global number of rows).  But when I pass the local_num_of_rows
> variable into the reinit function, this is the PETSC error I get:
>
> PETSC ERROR: Nonconforming object sizes
> PETSC ERROR: Sum of local lengths 512 does not equal global length 128, my
> local length 128
>
> Here is my reinit function and necessary vectors:
>
> std::vector<size_type> local_rows_per_process(num_procs,local_num_rows);
> std::vector<size_type> local_columns_per_process(num_procs,number_of_columns);
>
> spm.reinit(MPI_COMM_WORLD, dsp, local_rows_per_process,
> local_columns_per_process, my_proc);
>

> *The number of local rows for this example is local_num_rows=32, I printed to

> check.  Though when it is passed into the reinit function, it looks like it is

> using the global number of rows.*

>
> I get the same error from the constructor that doesn’t use the dynamic
> sparsity pattern:
>
> spm.reinit(MPI_COMM_WORLD,global_row_size,global_column_size,local_row_size,local_column_size,number_of_non_zeros);
> Just for clarifying this constructor, what is “local_rows” and “local_columns”
> for this constructor?  The documentation just says see the class
> documentation.  I see where the 4th constructor uses “local_rows_per_process”
> and this means how many rows do all other processors own and same for the
> columns so I thought I had that figured out for my constructor with the
> dynamic sparsity pattern but maybe not.  For this constructor, I just used the
> local number of rows and columns.
>
> Can someone please show me what they would do to debug this situation?

Can you come up with a small, self-contained test case that we can run to see
and debug what exactly you are doing?

Best
WB

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Zachary Streeter]

My project is in quantum scattering and I would like to have some operators be distributed PETSc objects.  So inside my OneBodyHamiltonianOperator class (for example), I would like to create a PETScWrappers::MPI::SparseMatrix and then use SLEPC to solve for the ground state and excited states. 

I tried to add in comments to everything to show my intent.

Here is the header file OneBodyHamiltonianOperator.h

-----------------------------------------------------------------------------------------

namespace quantumScattering {

class OneBodyHamiltonianOperator {
 public:
  /**
   * Declare type for container size.
   */
  using size_type = dealii::types::global_dof_index; // using dealii type for dealii types

  OneBodyHamiltonianOperator(const dealii::IndexSet &a_local_row_set,
                             const uint32_t a_my_proc,
                             const uint32_t a_num_procs);

  /// Destructor
  ~OneBodyHamiltonianOperator();

 private:
  dealii::PETScWrappers::MPI::SparseMatrix m_H1;
  dealii::DynamicSparsityPattern m_dynamic_sparsity_pattern; // want to use this for reinit for better performance
};

}  // namespace quantumScattering

Here is the OneBodyHamiltonianOperator.cc file

---------------------------------------------------------------------------------------------------

using namespace quantumScattering;

OneBodyHamiltonianOperator::OneBodyHamiltonianOperator(
    const dealii::IndexSet &a_local_row_set, const uint32_t a_my_proc,
    const uint32_t a_num_procs) {
  dealii::IndexSet local_owned(a_local_row_set.size()); // initialize IndexSet with the global row size
  local_owned.add_range(
      *a_local_row_set.begin(),
      *a_local_row_set.begin() + a_local_row_set.n_elements()); // add the number of local rows as range

  m_dynamic_sparsity_pattern.reinit(a_local_row_set.size(),
                                    a_local_row_set.size(), local_owned); // not used here but is the goal, need to get reinit working first...

  int guess = 50; // arbitrary guess for number of non-zeros

  // have a square matrix of size a_local_row_set.size() X a_local_row_set.size()

  // idea is to parallelize on the rows so the local rows should be of size a_local_row_set.n_elements()

  m_H1.reinit(MPI_COMM_WORLD, a_local_row_set.size(), a_local_row_set.size(),
              a_local_row_set.n_elements(), a_local_row_set.size(), guess); 
}

OneBodyHamiltonianOperator::~OneBodyHamiltonianOperator() {}

 

Here is a test::

----------------------------------------------------------------------------------------------------------------------------

#include "OneBodyHamiltonianOperator.h"

void
test(const int &n_proc, const int &my_proc, const ConditionalOStream &pcout)
{

  // arbitrary test variables. All operators are (nbas*nchannels) X (nbas*nchannels) and just parallelize the rows

  auto     nbas      = 64;
  auto     nchannels = 2;
  IndexSet local_row_set =
    Utilities::create_evenly_distributed_partitioning(my_proc,
                                                      n_proc,
                                                      nbas * nchannels); // set how all operators will be parallelized by rows

  OneBodyHamiltonianOperator H1(local_row_set, my_proc, n_proc);
}

int
main(int argc, char **argv)
{
  Utilities::MPI::MPI_InitFinalize mpi_initialization(argc, argv, 1);

  MPI_Comm   comm(MPI_COMM_WORLD);
  const auto my_proc = Utilities::MPI::this_mpi_process(comm);
  const auto n_proc  = Utilities::MPI::n_mpi_processes(comm);

  ConditionalOStream pcout(std::cout, (my_proc == 0));

  test(n_proc, my_proc, pcout);

  return 0;
}

This gave me the same error and it's fairly bare bones so I hope it is an easy fix.  The real case involves building a distributed dynamic sparsity matrix and using that to reinit the SparseMatrix.  Can I create a parallel sparse matrix using a distributed dynamic sparsity pattern or must I gather all elements of the dynamic sparsity pattern locally (that's a separate but important question)?

Please let me know if my test case is lacking in clarity etc. and I gravely appreciate the guidance!

Cheers,

Zachary

[Wolfgang Bangerth]

> My project is in quantum scattering and I would like to have some operators be
> distributed PETSc objects.  So inside my OneBodyHamiltonianOperator class (for
> example), I would like to create a PETScWrappers::MPI::SparseMatrix and then
> use SLEPC to solve for the ground state and excited states.
>
> I tried to add in comments to everything to show my intent.

Zachary -- can you put all of this into one single file and add all of the
necessary #includes at the top? Make it easy for us to have something we can
compile and run :-)

Cheers
W.

[Zachary Streeter]

#include <deal.II/base/conditional_ostream.h>
#include <deal.II/base/index_set.h>
#include <deal.II/base/mpi.h>

#include <deal.II/lac/dynamic_sparsity_pattern.h>
#include <deal.II/lac/petsc_sparse_matrix.h>

using namespace dealii;

class OneBodyHamiltonianOperator
{
public:
  /**
   * Declare type for container size.
   */
  using size_type = dealii::types::global_dof_index;

  OneBodyHamiltonianOperator(const dealii::IndexSet &a_local_row_set,
                             const uint32_t          a_my_proc,
                             const uint32_t          a_num_procs);

  /// Destructor
  ~OneBodyHamiltonianOperator();

private:
  dealii::PETScWrappers::MPI::SparseMatrix m_H1;
  dealii::DynamicSparsityPattern           m_dynamic_sparsity_pattern;

};


OneBodyHamiltonianOperator::OneBodyHamiltonianOperator(
  const dealii::IndexSet &a_local_row_set,
  const uint32_t          a_my_proc,
  const uint32_t          a_num_procs)
{
  dealii::IndexSet local_owned(a_local_row_set.size());

  local_owned.add_range(*a_local_row_set.begin(),
                        *a_local_row_set.begin() +
                          a_local_row_set.n_elements());

  m_dynamic_sparsity_pattern.reinit(a_local_row_set.size(),
                                    a_local_row_set.size(),
                                    local_owned);

  int guess = 50;

  m_H1.reinit(MPI_COMM_WORLD,
              a_local_row_set.size(),
              a_local_row_set.size(),
              a_local_row_set.n_elements(),
              a_local_row_set.size(),
              guess);
}

OneBodyHamiltonianOperator::~OneBodyHamiltonianOperator()
{}

void
test(const int &n_proc, const int &my_proc, const ConditionalOStream &pcout)
{

  auto     nbas      = 64;
  auto     nchannels = 2;
  IndexSet local_row_set =
    Utilities::create_evenly_distributed_partitioning(my_proc,
                                                      n_proc,
                                                      nbas * nchannels);

  OneBodyHamiltonianOperator H1(local_row_set, my_proc, n_proc);
}

int
main(int argc, char **argv)
{
  Utilities::MPI::MPI_InitFinalize mpi_initialization(argc, argv, 1);

  MPI_Comm   comm(MPI_COMM_WORLD);
  const auto my_proc = Utilities::MPI::this_mpi_process(comm);
  const auto n_proc  = Utilities::MPI::n_mpi_processes(comm);

  ConditionalOStream pcout(std::cout, (my_proc == 0));

  test(n_proc, my_proc, pcout);

  return 0;
}

 

Let me know if this is okay.  This compiled, ran, and produced the same error on my end.

Thank you!

Zachary

[Wolfgang Bangerth]

On 1/5/21 8:21 AM, Zachary Streeter wrote:
> Let me know if this is okay.  This compiled, ran, and produced the same error
> on my end.

Yes, that's the sort of testcase that makes it easy to debug :-)

In this call,

m_H1.reinit(MPI_COMM_WORLD,
a_local_row_set.size(),
a_local_row_set.size(),
a_local_row_set.n_elements(),
a_local_row_set.size(),
guess);

the 'local_columns' given in the fifth argument also need to add up to the
total number of columns. So if you change this to
m_H1.reinit(MPI_COMM_WORLD,
a_local_row_set.size(),
a_local_row_set.size(),
a_local_row_set.n_elements(),
a_local_row_set.n_elements(),
guess);
then things run fine.

That said, this specific function is deprecated:
https://dealii.org/developer/doxygen/deal.II/classPETScWrappers_1_1MPI_1_1SparseMatrix.html#a645ab9f99494a47ebb6492e92e707130
You will want to use the suggested alternative.

Best

[Zachary Streeter]

W.,

Ah okay I see, I will try that in my program and let you know.

Yes, I want to use the constructor with the dynamic sparsity pattern.  So with your suggestion in mind, would that just be the following:

dealii::IndexSet local_owned(a_local_row_set.size());

  local_owned.add_range(*a_local_row_set.begin(),
                        *a_local_row_set.begin() +
                          a_local_row_set.n_elements());

  m_dynamic_sparsity_pattern.reinit(a_local_row_set.size(),
                                    a_local_row_set.size(),
                                    local_owned);

std::vector<size_type> local_rows_per_process(num_procs, a_local_row_set.n_elements() );

std::vector<size_type> local_columns_per_process(num_procs, a_local_row_set.n_elements() ); // before used a_local_row_set.size() here!

m_H1.reinit(MPI_COMM_WORLD, m_dynamic_sparsity_pattern, local_rows_per_process, local_columns_per_process, my_proc);

[Wolfgang Bangerth]

On 1/5/21 11:32 AM, Zachary Streeter wrote:
>
> Yes, I want to use the constructor with the dynamic sparsity pattern.  So with
> your suggestion in mind, would that just be the following:
>
> dealii::IndexSet local_owned(a_local_row_set.size());
>   local_owned.add_range(*a_local_row_set.begin(),
>                         *a_local_row_set.begin() +
>                           a_local_row_set.n_elements());
>
>   m_dynamic_sparsity_pattern.reinit(a_local_row_set.size(),
>                                     a_local_row_set.size(),
>                                     local_owned);
>
> std::vector<size_type> local_rows_per_process(num_procs,
> a_local_row_set.n_elements() );

This creates a vector of size num_procs in which all entries are equal to
a_local_row_set.n_elements(). But why would all processors have the same
number of rows? And are you sure that a_local_row_set.n_elements() is the same
number on all processors anyway?

> std::vector<size_type> local_columns_per_process(num_procs,
> a_local_row_set.n_elements() ); // before used a_local_row_set.size() here!
>
> m_H1.reinit(MPI_COMM_WORLD, m_dynamic_sparsity_pattern,
> local_rows_per_process, local_columns_per_process, my_proc);

I would try to stick with the fourth reinit() function here:
https://dealii.org/developer/doxygen/deal.II/classPETScWrappers_1_1MPI_1_1SparseMatrix.html#a645ab9f99494a47ebb6492e92e707130
It's the simplest to use because every processor only has to pass an IndexSet
object that describes which rows (and columns) it owns.

[Zachary 42!]

W.,

Ah yes you are correct in your thinking. My test case was every process did have the same number of rows but I would need to communicate that with a gather to account for the number of rows not divisible by the number of given processors. I will use the 4th reinit function to avoid needing to do this, thank you!

Using the same number of local columns as local rows in the reinit functions is still puzzling to me, so let me be clear of my desired layout.

I have a matrix that is total_num_rows X total_num_columns. I would like to divide up, by processor, only on the row dimension so I would think the local layout would be ( total_num_rows / num_procs ) X total_num_columns, which is why I was trying to use size() for the local column number.

Here is my code again with hopefully better comments. Looking at the loop structure for the dynamic_sparsity_pattern should make it clear. I think the 3rd argument in the constructor for the dynamic sparsity pattern is wrong if I want these dimensions.

#include <deal.II/base/conditional_ostream.h>
#include <deal.II/base/index_set.h>
#include <deal.II/base/mpi.h>

#include <deal.II/lac/dynamic_sparsity_pattern.h>
#include <deal.II/lac/petsc_sparse_matrix.h>

using namespace dealii;

class OneBodyHamiltonianOperator
{
public:
/**
* Declare type for container size.
*/
using size_type = dealii::types::global_dof_index;

OneBodyHamiltonianOperator(const dealii::IndexSet &a_local_row_set,
const uint32_t a_my_proc,
const uint32_t a_num_procs);

/// Destructor
~OneBodyHamiltonianOperator();

private:
dealii::PETScWrappers::MPI::SparseMatrix m_H1;
dealii::DynamicSparsityPattern m_dynamic_sparsity_pattern;
};

OneBodyHamiltonianOperator::OneBodyHamiltonianOperator(
const dealii::IndexSet &a_local_row_set,
const uint32_t a_my_proc,
const uint32_t a_num_procs)
{

m_dynamic_sparsity_pattern.reinit(
a_local_row_set.size(),
a_local_row_set.size(), // sqaure matrix
a_local_row_set); // this maybe what's wrong for wanting
// a_local_row_set.n_elements() X a_local_row_set.size()

for (dealii::IndexSet::ElementIterator i = a_local_row_set.begin();
i !=
a_local_row_set.end(); // row dimension has a_local_row_set.n_elements()
// num of elements
++i)
{
for (uint32_t j = 0; j < a_local_row_set.size();
++j) // column dimension has a_local_row_set.size() dimension
{
auto ith = *i;
m_dynamic_sparsity_pattern.add(ith, j);
}
}

m_H1.reinit(a_local_row_set,
a_local_row_set, // would think number of local columns is the
// full dimension because only partitioning on
// row dimension
m_dynamic_sparsity_pattern,
MPI_COMM_WORLD);

for (dealii::IndexSet::ElementIterator i = a_local_row_set.begin();
i != a_local_row_set.end();
++i)
{
auto ith = *i;
for (uint32_t j = 0; j < m_dynamic_sparsity_pattern.row_length(ith); ++j)
{
m_H1.add(ith,
m_dynamic_sparsity_pattern.column_number(ith, j),
1); // added 1 for test value
}
}

m_H1.compress(dealii::VectorOperation::add);

}

OneBodyHamiltonianOperator::~OneBodyHamiltonianOperator()
{}

void
test(const int &n_proc, const int &my_proc, const ConditionalOStream &pcout)
{

// Have matrices with (nbas * nchannels) X (nbas * nchannels)

auto nbas = 64;
auto nchannels = 2;
IndexSet local_row_set = Utilities::create_evenly_distributed_partitioning(
my_proc,

n_proc,
nbas * nchannels); // partition on num_rows (i.e. nbas * nchannels)

OneBodyHamiltonianOperator H1(local_row_set, my_proc, n_proc);
}

int
main(int argc, char **argv)
{
Utilities::MPI::MPI_InitFinalize mpi_initialization(argc, argv, 1);

MPI_Comm comm(MPI_COMM_WORLD);
const auto my_proc = Utilities::MPI::this_mpi_process(comm);
const auto n_proc = Utilities::MPI::n_mpi_processes(comm);

ConditionalOStream pcout(std::cout, (my_proc == 0));

test(n_proc, my_proc, pcout);

return 0;
}

> --
> The deal.II project is located at http://www.dealii.org/
> For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
> --- You received this message because you are subscribed to a topic in the Google Groups "deal.II User Group" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/dealii/If0Ep9rJ67Q/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to dealii+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/d5d792d6-3c44-ccf4-903e-6825df57646e%40colostate.edu.

[Zachary 42!]

Sorry, I am not sure why my code’s format changed when I copied and pasted, trying again...

#include <deal.II/base/conditional_ostream.h>
#include <deal.II/base/index_set.h>
#include <deal.II/base/mpi.h>

#include <deal.II/lac/dynamic_sparsity_pattern.h>
#include <deal.II/lac/petsc_sparse_matrix.h>

using namespace dealii;

class OneBodyHamiltonianOperator
{
public:
/**
* Declare type for container size.
*/
using size_type = dealii::types::global_dof_index;

OneBodyHamiltonianOperator(const dealii::IndexSet &a_local_row_set,
const uint32_t a_my_proc,
const uint32_t a_num_procs);

/// Destructor
~OneBodyHamiltonianOperator();

private:
dealii::PETScWrappers::MPI::SparseMatrix m_H1;
dealii::DynamicSparsityPattern m_dynamic_sparsity_pattern;
};

OneBodyHamiltonianOperator::OneBodyHamiltonianOperator(
const dealii::IndexSet &a_local_row_set,
const uint32_t a_my_proc,
const uint32_t a_num_procs)
{

dealii::IndexSet local_owned_rows(a_local_row_set.size());
local_owned_rows.add_range(*a_local_row_set.begin(),
*a_local_row_set.begin() +
a_local_row_set.n_elements());
dealii::IndexSet local_owned_columns(a_local_row_set.size());
local_owned_columns.add_range(0, a_local_row_set.size());

m_dynamic_sparsity_pattern.reinit(a_local_row_set.size(),
a_local_row_set.size(),
local_owned_rows);

for (dealii::IndexSet::ElementIterator i = a_local_row_set.begin();
i != a_local_row_set.end();
++i)
{

for (uint32_t j = 0; j < a_local_row_set.size(); ++j)

{
auto ith = *i;
m_dynamic_sparsity_pattern.add(ith, j);
}
}

m_H1.reinit(local_owned_rows,
local_owned_rows,

m_dynamic_sparsity_pattern,
MPI_COMM_WORLD);

for (dealii::IndexSet::ElementIterator i = a_local_row_set.begin();
i != a_local_row_set.end();
++i)
{
auto ith = *i;
for (uint32_t j = 0; j < m_dynamic_sparsity_pattern.row_length(ith); ++j)
{
m_H1.add(ith,
m_dynamic_sparsity_pattern.column_number(ith, j),
1); // added 1 for test value
}
}

m_H1.compress(dealii::VectorOperation::add);
}

OneBodyHamiltonianOperator::~OneBodyHamiltonianOperator()
{}

void
test(const int &n_proc, const int &my_proc, const ConditionalOStream &pcout)
{
auto nbas = 64;
auto nchannels = 2;
IndexSet local_row_set =
Utilities::create_evenly_distributed_partitioning(my_proc,
n_proc,
nbas * nchannels);

OneBodyHamiltonianOperator H1(local_row_set, my_proc, n_proc);
}

int
main(int argc, char **argv)
{
Utilities::MPI::MPI_InitFinalize mpi_initialization(argc, argv, 1);

MPI_Comm comm(MPI_COMM_WORLD);
const auto my_proc = Utilities::MPI::this_mpi_process(comm);
const auto n_proc = Utilities::MPI::n_mpi_processes(comm);

ConditionalOStream pcout(std::cout, (my_proc == 0));

test(n_proc, my_proc, pcout);

return 0;
}

> On Jan 5, 2021, at 11:03 PM, Wolfgang Bangerth <bang...@colostate.edu> wrote:
>

[Wolfgang Bangerth]

On 1/6/21 8:07 AM, Zachary 42! wrote:
> Here is my code again with hopefully better comments. Looking at the loop structure for the dynamic_sparsity_pattern should make it clear. I think the 3rd argument in the constructor for the dynamic sparsity pattern is wrong if I want these dimensions.

Zachary -- there's no question in your email. What happens if you try with
your code?

Timo put it well: We enjoy helping, but there is a limit to how much time we
can offer.

[Zachary 42!]

Hi W.,

Apologies for not asking a specific question. Your help with the SparseMatrix constructor still left me confused and I was hoping for clarity. ( To answer your question, the code ran without error but the error came back once I tried to use Slepc. )

You said “the local number of columns also needs to add up to the global number of columns” but my desired layout is local_num_columns == global_num_columns. If I create the local dynamic sparsity pattern with dimension local_num_rows.n_elements() X local_num_rows.size(), does the sparse matrix constructor set the local_num_columns == global_num_columns? I don’t want to assume things but this is one of the pitfalls of abstraction, I suppose.

I only press this because I get the similar dimension error when I tried to use Slepc to solve the test matrix so maybe my initial sparse matrix dimension is wrong (or my matrix). Please take a look at the file I am attaching. It’s the same code you ran before but I added a section using Slepc to solve the eigenvalue problem. It’s bare bones and produces the dimension error just like before. This code works for one processor and given the error, this is not a surprise. For 2 processors, I get “Sum of local length 64 does not equal global length 128, my local length 64”.

I have been hitting this particular wall for quite some time but I know my last message Timo responded to did not come across like that at all. He is completely correct and I will keep his constructive comments in mind.

Much appreciated,

Zachary

[Wolfgang Bangerth]

> You said “the local number of columns also needs to add up to the global
> number of columns” but my desired layout is local_num_columns ==
> global_num_columns.

No, that's not what you want. You need to partition the columns in the same
way as you partition the vectors you want to multiply the matrix by. We
generally partition the vectors in the same way as the rows of the matrix, and
so then you also have to choose the same partitioning for the columns of the
matrix.

> I only press this because I get the similar dimension error when I tried to
> use Slepc to solve the test matrix so maybe my initial sparse matrix dimension
> is wrong (or my matrix).  Please take a look at the file I am attaching.  It’s
> the same code you ran before but I added a section using Slepc to solve the
> eigenvalue problem.  It’s bare bones and produces the dimension error just
> like before.  This code works for one processor and given the error, this is
> not a surprise.  For 2 processors, I get “Sum of local length 64 does not
> equal global length 128, my local length 64”.

If you look at the backtrace, at which point do you get this error? Is it
really out of the SLEPC functions? I would suspect that it comes out of the
set-up phase for your matrix.

[Zachary 42!]

No, that's not what you want. You need to partition the columns in the same way as you partition the vectors you want to multiply the matrix by. We generally partition the vectors in the same way as the rows of the matrix, and so then you also have to choose the same partitioning for the columns of the matrix.

Ah okay, I thought I wasn’t explaining my desired layout well.  I will change my matrix layout so I can have distributed square sparse matrices.

If you look at the backtrace, at which point do you get this error? Is it really out of the SLEPC functions? I would suspect that it comes out of the set-up phase for your matrix.

Yes, the error comes when I try to use the Slepc solver (the matrix builds without errors).  Running the attached file (Sorry, I’v been experimenting trying to get things working so I think I should include what I am testing now), At the bottom of the error output, I get:

----------------------------------------------------------------------------------------------------------------------------------------------------

An error occurred in Line <169> of file </Users/zachary/Documents/dealii/source/lac/slepc_solver.cc> in function

void dealii::SLEPcWrappers::SolverBase::solve(const unsigned int, unsigned int*)

The violated condition was:

ierr == 0

Additional information:

An error with error number 60 occurred while calling a SLEPc function

------------------------------------------------------------------------------------------------------------------------------------------------------

-Zachary

[Wolfgang Bangerth]

On 1/8/21 3:58 PM, Zachary 42! wrote:
>
> Yes, the error comes when I try to use the Slepc solver (the matrix builds
> without errors).  Running the attached file (Sorry, I’v been experimenting
> trying to get things working so I think I should include what I am testing
> now), At the bottom of the error output, I get:
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
> An error occurred in Line <169> of file
> </Users/zachary/Documents/dealii/source/lac/slepc_solver.cc> in function
> void dealii::SLEPcWrappers::SolverBase::solve(const unsigned int, unsigned int*)
> The violated condition was:
> ierr == 0
> Additional information:
> An error with error number 60 occurred while calling a SLEPc function
> ------------------------------------------------------------------------------------------------------------------------------------------------------

But that's a different error. Have you tried to look up what error code 60
means in SLEPc?

[Zachary 42!]

> But that's a different error. Have you tried to look up what error code 60 means in SLEPc?

Here is the entire error. I was interpreting the error was the “Nonconforming object size”, which is the error I was reporting. I will research the error 60 in Slepc, thank you!


[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Nonconforming object sizes
[1]PETSC ERROR: Sum of local lengths 64 does not equal global length 128, my local length 64
likely a call to VecSetSizes() or MatSetSizes() is wrong.
See https://www.mcs.anl.gov/petsc/documentation/faq.html#split
[1]PETSC ERROR: [0]PETSC ERROR: Nonconforming object sizes
[0]PETSC ERROR: Sum of local lengths 64 does not equal global length 128, my local length 64
likely a call to VecSetSizes() or MatSetSizes() is wrong.
See https://www.mcs.anl.gov/petsc/documentation/faq.html#split
[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.13.4, Aug 01, 2020
[0]PETSC ERROR: ./step-test_reinit.release on a named MasterChief.dhcp.lbl.gov by zachary Fri Jan 8 15:28:24 2021
[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.13.4, Aug 01, 2020
[1]PETSC ERROR: ./step-test_reinit.release on a named MasterChief.dhcp.lbl.gov by zachary Fri Jan 8 15:28:24 2021
[1]PETSC ERROR: Configure options --prefix=/opt/local/lib/petsc --with-environment-variables=1 --with-valgrind=0 --with-shared-libraries --with-debugging=1 --with-c2html-dir=/opt/local --with-x=0 --with-scalar-type=complex --with-clanguage=C++ COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp --with-blaslapack-lib="-L/opt/local/lib -lvecLibFort"
[1]PETSC ERROR: #1 PetscSplitOwnership() line 104 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_math_petsc/petsc/work/petsc-3.13.4/src/sys/utils/psplit.c
Configure options --prefix=/opt/local/lib/petsc --with-environment-variables=1 --with-valgrind=0 --with-shared-libraries --with-debugging=1 --with-c2html-dir=/opt/local --with-x=0 --with-scalar-type=complex --with-clanguage=C++ COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp --with-blaslapack-lib="-L/opt/local/lib -lvecLibFort"
[0]PETSC ERROR: #1 PetscSplitOwnership() line 104 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_math_petsc/petsc/work/petsc-3.13.4/src/sys/utils/psplit.c
[0]PETSC ERROR: #2 PetscLayoutSetUp() line 265 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_math_petsc/petsc/work/petsc-3.13.4/src/vec/is/utils/pmap.c
[0]PETSC ERROR: #3 MatCreate_Shell() line 1211 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_math_petsc/petsc/work/petsc-3.13.4/src/mat/impls/shell/shell.c
[0]PETSC ERROR: #4 MatSetType() line 96 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_math_petsc/petsc/work/petsc-3.13.4/src/mat/interface/matreg.c
[0]PETSC ERROR: #5 MatCreateShell() line 1329 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_math_petsc/petsc/work/petsc-3.13.4/src/mat/impls/shell/shell.c
[0]PETSC ERROR: #6 STGetOperator_Private() line 285 in /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_math_slepc/slepc/work/slepc-3.13.2/src/sys/classes/st/interface/stsolve.c
[1]PETSC ERROR: #2 PetscLayoutSetUp() line 265 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_math_petsc/petsc/work/petsc-3.13.4/src/vec/is/utils/pmap.c
[1]PETSC ERROR: #3 MatCreate_Shell() line 1211 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_math_petsc/petsc/work/petsc-3.13.4/src/mat/impls/shell/shell.c
[1]PETSC ERROR: #4 MatSetType() line 96 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_math_petsc/petsc/work/petsc-3.13.4/src/mat/interface/matreg.c
[1]PETSC ERROR: #5 MatCreateShell() line 1329 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_math_petsc/petsc/work/petsc-3.13.4/src/mat/impls/shell/shell.c
[1]PETSC ERROR: [0]PETSC ERROR: #7 STApply() line 64 in /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_math_slepc/slepc/work/slepc-3.13.2/src/sys/classes/st/interface/stsolve.c
[0]PETSC ERROR: #8 EPSGetStartVector() line 809 in /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_math_slepc/slepc/work/slepc-3.13.2/src/eps/interface/epssolve.c
#6 STGetOperator_Private() line 285 in /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_math_slepc/slepc/work/slepc-3.13.2/src/sys/classes/st/interface/stsolve.c
[1]PETSC ERROR: #7 STApply() line 64 in /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_math_slepc/slepc/work/slepc-3.13.2/src/sys/classes/st/interface/stsolve.c
[1]PETSC ERROR: #8 EPSGetStartVector() line 809 in /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_math_slepc/slepc/work/slepc-3.13.2/src/eps/interface/epssolve.c
[1]PETSC ERROR: [0]PETSC ERROR: #9 EPSSolve_KrylovSchur_Symm() line 32 in /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_math_slepc/slepc/work/slepc-3.13.2/src/eps/impls/krylov/krylovschur/ks-symm.c
[0]PETSC ERROR: #10 EPSSolve() line 149 in /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_math_slepc/slepc/work/slepc-3.13.2/src/eps/interface/epssolve.c
#9 EPSSolve_KrylovSchur_Symm() line 32 in /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_math_slepc/slepc/work/slepc-3.13.2/src/eps/impls/krylov/krylovschur/ks-symm.c
[1]PETSC ERROR: #10 EPSSolve() line 149 in /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_math_slepc/slepc/work/slepc-3.13.2/src/eps/interface/epssolve.c
libc++abi.dylib: terminating with uncaught exception of type dealii::SLEPcWrappers::SolverBase::ExcSLEPcError:
--------------------------------------------------------
An error occurred in line <169> of file </Users/zachary/Documents/dealii/source/lac/slepc_solver.cc> in function

void dealii::SLEPcWrappers::SolverBase::solve(const unsigned int, unsigned int *)
The violated condition was:
ierr == 0
Additional information:
An error with error number 60 occurred while calling a SLEPc function
--------------------------------------------------------

libc++abi.dylib: terminating with uncaught exception of type dealii::SLEPcWrappers::SolverBase::ExcSLEPcError:
--------------------------------------------------------
An error occurred in line <169> of file </Users/zachary/Documents/dealii/source/lac/slepc_solver.cc> in function

void dealii::SLEPcWrappers::SolverBase::solve(const unsigned int, unsigned int *)
The violated condition was:
ierr == 0
Additional information:
An error with error number 60 occurred while calling a SLEPc function

————————————————————————————

-Zachary

> --
> The deal.II project is located at http://www.dealii.org/
> For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
> --- You received this message because you are subscribed to a topic in the Google Groups "deal.II User Group" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/dealii/If0Ep9rJ67Q/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to dealii+un...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/a3b74574-5a16-47d8-c69c-ce52243f67d0%40colostate.edu.