Optimization Tutorial Not Working As Expected

[Rassam Yazdi]

Hello,

I am hoping to use SUAVE as a means of validating another conceptual design tool's fuel burn estimates for slowing down the cruise Mach of a regional jet by 10%. 

I was going through the optimization tutorial when I tried to do a run and I ended up with the following string of values:

Inputs

[0.92 0.8 ]

Obj

[0.73230996]

Inputs

[0.92000001 0.8       ]

Obj

[0.73231002]

Inputs

[0.92       0.80000001]

Obj

[0.73230996]

Inputs

[0.5        0.92612154]

Obj

[0.83769962]

Inputs

[0.72098521 0.85976202]

Obj

[0.7430164]

Inputs

[0.82165641 0.82953153]

Obj

[0.73179085]

Inputs

[0.87077104 0.81478293]

Obj

[0.73083287]

Inputs

[0.89522198 0.80744058]

Obj

[0.73128146]

Inputs

[0.90749683 0.80375457]

Obj

[0.73172111]

Inputs

[0.91368297 0.80189694]

Obj

[0.73199522]

Inputs

[0.91680648 0.80095898]

Obj

[0.73214646]

Inputs

[0.91838505 0.80048495]

Obj

[0.73222626]

Inputs

[0.91918321 0.80024527]

Obj

[0.73226722]

Inputs

[0.91958685 0.80012406]

Obj

[0.73228826]

Inputs

[0.91958687 0.80012406]

Obj

[0.73228825]

Inputs

[0.91958685 0.80012408]

Obj

[0.73228833]

Inputs

[0.75023011 0.6       ]

Obj

[0.76307601]

Inputs

[0.83829658 0.70406566]

Obj

[0.74508974]

Inputs

[0.88035806 0.75376851]

Obj

[0.73800285]

Inputs

[0.90060637 0.77769537]

Obj

[0.73494427]

Inputs

[0.91039161 0.78925831]

Obj

[0.73354944]

Inputs

[0.9151294  0.79485682]

Obj

[0.73289365]

Inputs

[0.91742543 0.79756997]

Obj

[0.73258037]

Inputs

[0.91853862 0.7988854 ]

Obj

[0.73242969]

Inputs

[0.91907846 0.79952331]

Obj

[0.73235673]

Inputs

[0.91934027 0.79983269]

Obj

[0.73232164]

Inputs

[0.91946727 0.79998276]

Obj

[0.73230433]

Inputs

[0.91946729 0.79998276]

Obj

[0.73230426]

Inputs

[0.91946727 0.79998278]

Obj

[0.73230433]

Inputs

[0.91968017 0.79992405]

Obj

[0.73231481]

Inputs

[0.91957264 0.7999537 ]

Segment did not converge. Segment Tag: cruise

Error Message:

The iteration is not making good progress, as measured by the

  improvement from the last ten iterations.

Obj

[0.7323096]

Inputs

[0.91951942 0.79996838]

Obj

[0.7323071]

Inputs

[0.91949306 0.79997565]

Obj

[0.73230578]

Inputs

[0.91948002 0.79997925]

Obj

[0.73230502]

Inputs

[0.91947357 0.79998102]

Obj

[0.73230468]

Inputs

[0.91947039 0.7999819 ]

Obj

[0.73230466]

Inputs

[0.9194688  0.79998234]

Obj

[0.73230444]

Inputs

[0.91946802 0.79998255]

Obj

[0.73230436]

Inputs

[0.91946764 0.79998266]

Obj

[0.73230431]

Inputs

[0.91946746 0.79998271]

Obj

[0.73230456]

Optimization terminated successfully    (Exit mode 0)

            Current function value: [0.73230456]

            Iterations: 3

            Function evaluations: 40

            Gradient evaluations: 3

[0.91946746 0.79998271]

fuel burn =  [7323.04558952]

fuel margin =  [1.66407523]

Is this the expected output? The Fuel Burn looks more or less unchanged from where it started from. Perhaps too few design variables are free for the optimizer to exploit.

On the same note, is there a comprehensive list of all viable design variable names that could be used in the problem.inputs array? 

Ideally, I plan on having span, sweep, chord, thickness, altitude (already free in the tutorial), and max thrust free in an optimization run to evaluate the gains associated with dropping cruise speed to M 0.7 instead of M 0.78.

Thanks in advance,

Rassam Yazdi

[ebot...@gmail.com]

Hi Rassam,

Well this is annoying problem we've been having with SLSQP. So the results you're seeing are to be expected, but that's not correct. If you do the variable sweep from the tutorials (I refined a ton here) you'll see this:

As you can see SUAVE creates a nice smooth convex design space, with the optimum at the top right. But for whatever reason SLSQP cannot do anything with it. I've searched for bugs and made sure that when you zoom in it's still behaving smoothly, yet SLSQP still does not work. In the tutorial we use SLSQP because everyone who has SUAVE on their computer has SLSQP installed (and it has worked in some cases for us).

I suggest for your project you use another optimizer, I suggest taking a look at PyOptSparse. Internally we use SNOPT with PyOptSparse for all of our optimization problems. Your intentions for your future optimization problem including design variables makes sense and should be no issue.

One last thing, I'm going to apologize, let me get on my soap box about what you said at the top. Some definitions in my own words:

• Validation is the process of making sure the code is valid to real world conditions (think rigorous).
  
• Verification is a check to verify that the code is functioning properly (think quick).

You're looking to verify that this other code is functioning properly (i.e free of major bugs). Any decent conceptual design code is said to be good when it's +/- 5% to reality (although SUAVE's quite good). So comparing two design codes could lead you to be +/- 10% off of reality!

Ok, off my soapbox. Again I apologize.

Let us know if you have any more questions.

-Emilio

[ebot...@gmail.com]

Hi Rassam,

I have some more information. I remembered we've seen issues with this before. SUAVE is behaving correctly but the signs of the constraints are incorrect in SLSQP.

If you take a look at the documentation here for SLSQP, you'll see that the inequality constraints are satisfied when they are above 0. This is satisfied with the tutorials in SUAVE.

However, I reran the tutorial and only changed the scale to -1 for the constraints. This converged as it should. So either SLSQP's documentation or algorithm is wrong.

-Emilio

[Rassam Yazdi]

You are indeed correct! It looks to be reducing the fuel now. 

And yes, I believe the word I was looking for is along the lines of verification. I am a summer student working in Prof. Zingg's lab at UTIAS and I've been using a PhD student's conceptual design tool to evaluate the fuel benefits of slowing down regional and single isle aircraft from M 0.78 to M 0.7. Our tool is indicating reductions on the order of 3-4% so I wanted to see if SUAVE produces similar results.

I have three questions regarding this: 

1) I see that there are tags for design variables to be included in the optimization (see attached image), is there a document or something like that for highlighting what all possible tags are? I am not sure if I would be able to guess the names of such design variables as they may not follow what I see in missions.py and vehicle.py. Specifically, I am looking to have the following variables free:

• cruise altitude
• chord
• span
• sweep
• thickness
• altitude
• thrust (this one seems to be reliant on a prespecified altitude and cruise mach as seen in the "design sizing conditions" section of vehicle.py, would that imply that it cannot be a design variable?)

2) How long should I expect the optimization to take? 

3) Is a full vehicle and mission breakdown reported anywhere (i.e. new MTOW, L/D, etc)?

Thank you for taking the time to respond to my (rather long) questions!

Sincerely,

Rassam

[ebot...@gmail.com]

Rassam,

1. There are so many possible tags, we couldn't never make a list. I suggest you put some print statements (or if you have an IDE with a debugger some stops) to see what's in the data structure.

The idea with the optimization framework is that you pick an alias name for the variable you want to use. Then you correspond that alias name with the name of the variable in the data structure. The exact routing for your problem will always be different from another users. Some SUAVE users are working on quadcopters and some people are working on flying wings, so the data is quite different.

2. It depends on your computational resources, the optimizer used, your bounds, the number of variables, and the types of variables. Could be a couple minutes or could be a couple days. I highly suggest if you have a multicore machine you take a look at pyOptsparse and install mpi and mpi4py. That will execute multiple SUAVE processes at once.

3. If you look at the tutorials we have a weight_breakdown function that calculates that. However, MTOW is an input not an output. I suggest you add MTOW to your optimization process as a variable (or leave it fixed). We don't save L/D explicitly, if you look at the plots we're just taking the CL and dividing by CD. I suggest you dive into the results that are generated, SUAVE makes a lot of results for a mission and they're all there.

-Emilio

PS. Not necessarily SUAVE related, but aircraft design is driven by constraints. Having a realistic set of constraints in your design problem is the difference between a reasonable and unreasonable design.

[Rassam Yazdi]

Great, thank you for answering all my questions, Emilio. I really appreciate it.

I do have certain constraints for things like wing Yehudi and area so hopefully I can implement them correctly in SUAVE.

Sincerely,

Rassam 

[Jeswin Joseph]

Hi Emilio & Rassam,

I recently installed SUAVE 2.5.2 and python IDE v3.9.xx. 

I ran the regional jet optimization tutorial by only changing the constraint scale to -1.0 but my iterations are converging close to the input space edge rather than the minimal fuel burn points. 

Scaling to 1.0, I get the solution either converged very close to initial input values or it converges close to the edge of space.

I see that you have solved this issue in this thread.

Is there some work I'm missing to get the optimizer converged? Can you help me with get it converged?

Best,

Jeswin

On Monday, August 9, 2021 at 2:52:20 PM UTC-4 ebot...@gmail.com wrote:

[Feng Huang]

Hi Jeswin Joseph,

I have met the same problem. Have you solved it?

Thanks in advance,

Feng Huang

[Feng Huang]

Dear Emilio

Is there any update on the issue of optimizations not working correctly? I'm having the same problem at the moment and I'm stuck here. I have chaged the scale to -1.0, but the iteration cannot converge.

Very thanks,

Huang Feng