Automation of Copasi

50 views
Skip to first unread message

Harry Wilson

unread,
Feb 28, 2025, 4:04:58 AMFeb 28
to COPASI User Forum
Hello,
I am trying to figure out the mechanism of an inorganic nanocluster synthesis. For this I have the experimental dat and I need to try it with multiple mechanisms.(at least more than 50 mechanisms). And I also have different experimental conditions and doing it one by one is tedious. Is it possible for me to automate this in Jupiter lab. My aim is to submit the cps files with different mechanism made from the copasi gui and the experimental data in Jupyter lab so that it can run the parameter estimation for all the input files I give. 

I have tried with the copasi basico, but it is not able to detect the experimnet data. When I try to upgrade basico I am getting version 0.3 and it does not have load_model option as well. Can you kindly let me know if this is possible and which version to use copasi and if it is can  you provide me a sample code for parameter estimation. Thank you for your attention to this matter.
Kind regards, 
Harry

Frank Bergmann

unread,
Feb 28, 2025, 4:30:31 AMFeb 28
to COPASI User Forum
Hello Harry, 

unfortunately there was a basico package before in pypi, so the name there is copasi-basico. Please install that: 

pip install copasi-basico

as for an example for parameter estimation see for example: 


best
Frank

Harry Wilson

unread,
Mar 3, 2025, 5:36:44 AMMar 3
to COPASI User Forum
Dear Frank,

When trying with copasi-basico, it is not able to get the experiment data from the cps file. Do you know how to resolve this issue? Thank you for your attention.
Best,
Harry

Frank Bergmann

unread,
Mar 3, 2025, 5:50:48 AMMar 3
to COPASI User Forum
Hello Harry, 

it entirely depends on how you are calling basico and where your experimental data files are (are you running it locally, or in colab? are you in the folder with the data). I recommend saving the data files next to the cps file. And then they ought to be able to be read in just fine. Alternatively you can export a combine archive from the CopasiUI that will include the model and the data files and you'll be able to load that into basico (using the same load_model command). 

If you have any specific issues, feel free to mail me directly, or if you think it is a bug in basico you could file it directly in https://github.com/copasi/basico/issues, where you will see how I resolve the issue. 

best
Frank

Harry Wilson

unread,
Mar 3, 2025, 9:05:10 AMMar 3
to COPASI User Forum
Dear Frank,

I have tried doing this but I do not know why I am not getting the values for the rate constants (parameters I am estimating). I get the values of them from CopasiUI, but from basico I am getting NAN as result. Do you know any potential reason for so? I will be attaching my cps and txt file for your reference. Thank you

Best,
Harry
no_x.cps
Screenshot 2025-03-03 150316.png
kinetics_with_intermediate_min.txt

Frank Bergmann

unread,
Mar 3, 2025, 9:20:21 AMMar 3
to COPASI User Forum
I've made one modification to the cps file, since it tried to get the experimental data from the subfolder 'text files' which did not exist on my machine. Apart from that it is working directly from me. 

Since you get 'NaN' values for the solution it seemed parameter estimation failed for you. 

I'll attach the changed files along with a screenshot that shows how i called it. 

best
Frank
no_x_changed.cps
screenshot_pe_basico.png
kinetics_with_intermediate_min.txt

Harry Wilson

unread,
Mar 3, 2025, 11:05:17 AMMar 3
to COPASI User Forum
Dear Frank, 
Thank you for your reply. I was able to run it with different experimental data at the same time. Is it possible to get the standard deviation of the rate constants estimated in basico like we get in CopasiUI.
Thanking you,
Harry

Frank Bergmann

unread,
Mar 3, 2025, 11:13:29 AMMar 3
to COPASI User Forum
Indeed, you simply run `get_fit_statistic(include_parameters=True)` this will return you the fit statistic as dictionary. In the parameters dictionary you will find the std_dev as displayed in the GUI. 

best
Frank

below an example output:

```
>>> get_fit_statistic(include_parameters=True)
... {'obj': 0.000716186200998224, 'rms': 0.004231389254719494, 'sd': 0.004399589050213986, 'f_evals': 8, 'failed_evals_exception': 0, 'failed_evals_nan': 0, 'constraint_evals': 0, 'failed_constraint_evals': 0, 'cpu_time': 0.0, 'data_points': 66, 'valid_data_points': 40, 'evals_per_sec': 0.0, 'parameters': [{'name': '(R1).k1', 'lower': 1e-06, 'start': 0.036116164940491434, 'value': 0.036116164940491434, 'upper': 1e06, 'std_dev': 0.011820603686779452, 'coeff_of_variation': 32.729398888991255, 'gradient': 3.919302741350599e-05}, {'name': '(R2).k1', 'lower': 1e-06, 'start': 0.00016963980982578393, 'value': 0.00016963980982578393, 'upper': 1e06, 'std_dev': 1.1394697894687698e-05, 'coeff_of_variation': 6.716995206720512, 'gradient': 0.02880551197358067}, {'name': '(R3).k1', 'lower': 1e-06, 'start': 0.10422701110816807, 'value': 0.10422701110816807, 'upper': 1e06, 'std_dev': 0.008797609679571534, 'coeff_of_variation': 8.440815471952147, 'gradient': 2.989119384221543e-05}]}
```

Harry Wilson

unread,
Mar 3, 2025, 12:22:39 PMMar 3
to COPASI User Forum
I was trying to see the reproducibility and I was getting different rate constants when I change the order of experimental data. 
I will explain the scenario: I used two different experimental data in the same cps model but depending on which experimental data I ran first, I am getting varied parameters. Would you know how to prevent this scenario?
Thanking you,
Harry

Screenshot 2025-03-03 181838.png
Screenshot 2025-03-03 181814.png

Frank Bergmann

unread,
Mar 3, 2025, 12:46:14 PMMar 3
to COPASI User Forum
We would have to see the python code you were using to confirm. Otherwise it is difficult to say what happened. One guess is that you run with update_model=True, in which case running the first parameter estimation influences the run of the 2nd one. Another guess is that the parameter is not uniquely identifiable, you might want to look into that.

I'm happy to have a look if you send me the code and additional data file. 

best
Frank

Harry Wilson

unread,
Mar 3, 2025, 1:10:01 PMMar 3
to COPASI User Forum
This is the code:
import os
import pandas as pd
from basico import *


model_path = "/home/wilson0003/Copasi/no_x.cps"


experiment_files = [
    "/home/wilson0003/Copasi/text files/kinetics_with_intermediate_min.txt",
    "/home/wilson0003/Copasi/text files/Kinetics.txt",
   ]


all_results = {}


for i, exp_file in enumerate(experiment_files):
    exp_name = f"exp_{i+1}"  # Unique name for each experiment


    if not os.path.exists(exp_file):
        print(f"⚠️ Skipping {exp_file} - File not found!")
        continue

    print(f"\n🔹 Using experiment file: {exp_file}")

   
    model = load_model(model_path)

   
    exp_data = pd.read_csv(exp_file, sep="\t")  # Ensure correct separator
    print(f"First rows of {exp_name} experimental data:\n", exp_data.head())

   
    if exp_name in get_experiment_names():
        remove_experiment(exp_name)

 
    add_experiment(exp_name, exp_data)
    print(f"✅ Experiment {exp_name} added successfully!")

   
    mapping = get_experiment_mapping(exp_name)
    mapping["cn"] = mapping["cn"].str.replace("ParticleNumber", "Concentration")

   
    print(f"🔹 Running Parameter Estimation for {exp_name}...")
    run_parameter_estimation(method="Particle Swarm", update_model=False)  # 🔹 Prevents model contamination

    fit_stats = get_fit_statistic(include_parameters=True)

   
    parameters_with_sd = []
    for param in fit_stats.get("parameters", []):
        parameters_with_sd.append({
            "name": param["name"],
            "value": param["value"],
            "std_dev": param.get("std_dev", "N/A"),  # Get SD if available
            "coeff_of_variation": param.get("coeff_of_variation", "N/A")
        })

   
    result_fit = pd.DataFrame(parameters_with_sd)
    print(f"📊 Fitted Parameters for {exp_name}:\n", result_fit)

   
    all_results[exp_name] = result_fit

   
    plot_per_dependent_variable()


all_results_df = pd.concat(all_results, names=["Experiment"])
all_results_df.to_csv("/home/wilson0003/Copasi/parameter_estimation_results_with_sd.csv")
print("\n✅ All parameter estimations completed & saved!")

End of code.

I will attach both the experiment data. One data is with the concentration of intermediate and one is without that data. Now if I save the cps file with either of these experiment data and run this code for changing the experiment data automatically, i am not getting same results. Thank you very much for taking your time to look ino this. I am trying this so that I can run this for 200 different scenarios.
Thanking you,
Harry

kinetics_with_intermediate_min.txt
Kinetics.txt
no_x.cps

Frank Bergmann

unread,
Mar 3, 2025, 1:48:57 PMMar 3
to COPASI User Forum
I would remove all experiments using 'remove_experiments()' after the call to load. Otherwise you'll leave the experiment which is named 'Experiment' in and this will change your results a bit in the first place. 

Also, you might want to consider using a local method (like LM: `run_parameter_estimation(method=PE.LEVENBERG_MARQUARDT, update_model=False)  # 🔹 Prevents model contamination`). That is more likely to run the along the same trajectory. Particle Swarm creates a swarm randomly positioned in your parameter space, and since you are not controlling the seed I would be surprised if you got precisely the same results every time you run. 

I hope this helps, 
best
Frank
 

Harry Wilson

unread,
Mar 5, 2025, 9:58:38 AMMar 5
to COPASI User Forum
Dear Frank,
While trying to optimise the code I started encountering a new problem. Some of the cps files i am using in the program gives me different parameters (rate constants) each time I run. I am using particle swarm method. Some variation in the value would have been ok, but the order of magnitude itself is varying. Do you know what might be the probable cause? Or is there a minimum requirement of how many times I have to run it to get fixed values?
this is the code i am using:
import os
import pandas as pd
from basico import *

# ✅ Folder paths
cps_folder = "/home/wilson0003/Copasi/x"  # Folder containing CPS files (without pre-linked experiments)
exp_folder = "/home/wilson0003/Copasi/y"  # Folder containing experimental data files

# ✅ List all CPS models
cps_files = [os.path.join(cps_folder, f) for f in os.listdir(cps_folder) if f.endswith(".cps")]
experiment_files = [os.path.join(exp_folder, f) for f in os.listdir(exp_folder) if f.endswith(".txt")]

# ✅ Iterate over CPS models
for cps_file in cps_files:
    print(f"\n🔹 Processing CPS Model: {cps_file}")

    # ✅ Load model
    model = load_model(cps_file)
    if model is None:
        print(f"❌ Failed to load CPS model: {cps_file}")
        continue

    # ✅ Remove any pre-existing experiments (Ensures a clean slate)
    existing_experiments = get_experiment_names()
    for exp in existing_experiments:
        remove_experiment(exp)
    print("✅ Cleared any pre-existing experiments.")

    # ✅ Iterate over experimental datasets
    for exp_file in experiment_files:
        exp_name = os.path.basename(exp_file).replace(".txt", "")

        print(f"\n✅ Loading Experimental Data: {exp_file}")

        exp_data = pd.read_csv(exp_file, sep="\t")  # Ensure correct separator
        if exp_data.empty:
            print(f"❌ Experimental data {exp_file} is empty! Skipping.")
            continue

        # ✅ Add new experiment

        add_experiment(exp_name, exp_data)
        print(f"✅ Experiment {exp_name} added successfully!")

        # ✅ Ensure the experiment was applied correctly
        if exp_name not in get_experiment_names():
            print(f"❌ Failed to apply experiment: {exp_name}")
            continue
        mapping = get_experiment_mapping(exp_name)
        print(mapping)

        # ✅ Run parameter estimation using "Levenberg - Marquardt"

        print(f"🔹 Running Parameter Estimation for {exp_name}...")
        run_parameter_estimation(method=PE.PARTICLE_SWARM, update_model=True)

        # ✅ Extract standard deviations from fit statistics
        fit_stats = get_fit_statistic(include_parameters=True)

        # ✅ Extract parameters with standard deviation

        parameters_with_sd = []
        for param in fit_stats.get("parameters", []):
            parameters_with_sd.append({
                "cps_model": os.path.basename(cps_file),
                "experiment": os.path.basename(exp_file),

                "name": param["name"],
                "value": param["value"],
                "std_dev": param.get("std_dev", "N/A"),
                "coeff_of_variation": param.get("coeff_of_variation", "N/A")
            })
        print("Saving parameters:", parameters_with_sd)
        # ✅ Save results
        df = pd.DataFrame(parameters_with_sd)
        df.to_csv(f"{cps_file}_{exp_name}.csv", index=False)
        print(f"✅ Results saved: {cps_file}_{exp_name}.csv")

        # ✅ Remove experiment after use (Prevents Memory Issues)
        remove_experiments()
        print(f"✅ Removed experiment: {exp_name}")

print("\n✅ All CPS models processed with all experimental data!")

I can also give you the problematic cps and experiment data. Thank you for you help in this matter.

Best,
Harry
kinetics_with_intermediate_min.txt
no_z.cps

Pedro Mendes

unread,
Mar 5, 2025, 11:18:04 AMMar 5
to copasi-u...@googlegroups.com
*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***

Dear Harry,

minimization with global (stochastic) algorithms is like this. You don't
always get the same result even if starting from the same initial
position. If you are getting very different results on each run, then
that is an indication that you did not run the algorithm long enough. In
particle swarm you can change this with the parameter "Iteration Limit".

This is not specific to BasiCO, it is the same thing in the GUI. It is
the same thing with any software that runs one of this type of
optimizers too.

By the way, paramater estimation (and optimization) is a task that is
much better used in an interactive/exploratory way, which is much easier
to do on a GUI than programmatically... I personally would not expect to
solve a parameter estimation problem on the first trial run, I would run
several times, adjusting settings between them, and trying several
algorithms. In the end, it is not the average of several runs that you
want, you actually want the one that gave you the lowest value of the
objective function.

Hope this helps
Pedro
> <https://github.com/copasi/basico/
> issues>, where you will see how I
> <https://
> basico.readthedocs.io/en/
> latest/notebooks/
> Setting_up_Parameter_Estimation.html>
> --
> You received this message because you are subscribed to the Google
> Groups "COPASI User Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to copasi-user-fo...@googlegroups.com
> <mailto:copasi-user-fo...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/copasi-
> user-forum/7c893b3f-da03-4532-bb03-a4accbbd08bdn%40googlegroups.com
> <https://groups.google.com/d/msgid/copasi-user-forum/7c893b3f-da03-4532-
> bb03-a4accbbd08bdn%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Pedro Mendes, PhD
Professor and Director
Richard D. Berlin Center for Cell Analysis and Modeling
University of Connecticut School of Medicine
group website: http://www.comp-sys-bio.org

Harry Wilson

unread,
Mar 5, 2025, 11:28:32 AMMar 5
to COPASI User Forum
Dear Pedro,
I ran the code several times and like you suggested the parameters started to vary very less. But the values I got from GUI and basico are different. Do you know what might be the problem? I made sure that th mapping and the optimization methodologies are same in both gui and basico. 
I am trying programing because i need to find the most suitable mechanism and i already have more than 80 cps files with different reaction pathways and 5 experimental data. So doing this in gui can be challenging. Thank you for your insights.
Best,
Harry

Pedro Mendes

unread,
Mar 5, 2025, 12:26:24 PMMar 5
to copasi-u...@googlegroups.com
*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***

Of course, it could be some problem with the parameter mapping or the
settings. If you have .cps files created with the GUI then you could try
running them from BasiCO simply by

load_model(file.cps)

and then straight to running the parameter estimation task

I would compare the output of that with the output of the file you
constructed from scratch within BasiCO (assuming that is what you did).

Of course, you also need to run the GUI constructed file several times
to check if you get such a wide variety of results.

Pedro
> > github.com/copasi/basico/issues <http://github.com/copasi/basico/
> issues>
> > <https://github.com/copasi/basico/ <https://github.com/copasi/
> basico/>
> > basico.readthedocs.io/en/ <http://basico.readthedocs.io/en/>
> copasi- <https://groups.google.com/d/msgid/copasi->
> > user-forum/7c893b3f-da03-4532-bb03-
> a4accbbd08bdn%40googlegroups.com <http://40googlegroups.com>
> > <https://groups.google.com/d/msgid/copasi-user-forum/7c893b3f-
> da03-4532- <https://groups.google.com/d/msgid/copasi-user-
> forum/7c893b3f-da03-4532->
> > bb03-a4accbbd08bdn%40googlegroups.com?
> utm_medium=email&utm_source=footer <http://40googlegroups.com?
> utm_medium=email&utm_source=footer>>.
>
> --
> Pedro Mendes, PhD
> Professor and Director
> Richard D. Berlin Center for Cell Analysis and Modeling
> University of Connecticut School of Medicine
> group website: http://www.comp-sys-bio.org <http://www.comp-sys-
> bio.org>
>
> --
> You received this message because you are subscribed to the Google
> Groups "COPASI User Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to copasi-user-fo...@googlegroups.com
> <mailto:copasi-user-fo...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/copasi-
> user-forum/051b329b-b6b7-4f48-b4a2-3be73658e6fdn%40googlegroups.com
> <https://groups.google.com/d/msgid/copasi-user-forum/051b329b-b6b7-4f48-
> b4a2-3be73658e6fdn%40googlegroups.com?utm_medium=email&utm_source=footer>.
Reply all
Reply to author
Forward
0 new messages