Gcode Simplification programs (Simplify3D Workaround)

1,194 views
Skip to first unread message

Leon Grossman

unread,
Oct 21, 2015, 8:47:59 AM10/21/15
to Smoothieware Support
This has now reached the point of needing it's own topic. Whether it's to avoid the dreaded crash or to improve your performance of USB printing, these apps are for you.

I wanted to learn swift so I made this app. https://github.com/leonmf/S3DFix

Here's my comments from the latest commit. I'm very, very new to swift and I, obviously don't yet have a handle on threading for UI responsiveness and performance tuning is not yet something I know how to do. The hasPrefix function seems to be a big performance killer so I'll probably replace that with a substring from 0 - 2 for the absolute/relative test.

- XY Resolution and E Resolution are not on the UI yet.
- UI Locks up when operating (need to add threading?)
- Performance is very, very, very slow

On the upside, it works.

Mikk Kiilaspää posted a web page which does the same thing here: http://mikk36.eu/SimplifyS3D/
The source code is here: https://github.com/Mikk36/SimplifyS3D

Walter Hsiao Provided us with this initial code that both Mikk and I worked from

#include <iostream>

#include <fstream>

#include <math.h>


using namespace std;


#define RESOLUTION_THRESHOLD .01f

#define EXTRUSION_THRESHOLD .0005f


string removeWhitespace(string s) {

    string::iterator end_pos = remove(s.begin(), s.end(), ' ');

    s.erase(end_pos, s.end());

    return s;

}


bool startsWith(const string &s, const string &prefix) {

    return (s.length() >= s.length() && s.find(prefix) == 0);

}



float getParameter(const string &s, char c, bool * found = NULL) {

    size_t i = s.find(c);

    if (i != string::npos) {

        if (found) *found = true;

        size_t n = s.find_first_not_of("-0123456789.", i+1);

        string valstring = s.substr(i+1, n-i-1);

        return atof(valstring.c_str());

    } else {

        if (found) *found = false;

        return 0;

    }

}



float xyDistance(const string &a, const string &b) {

    bool found = true;

    

    float aX = getParameter(a, 'X', &found);

    if (!found) return -1;

    

    float aY = getParameter(a, 'Y', &found);

    if (!found) return -1;

    

    float bX = getParameter(b, 'X', &found);

    if (!found) return -1;

    

    float bY = getParameter(b, 'Y', &found);

    if (!found) return -1;

    

    float dX = bX - aX;

    float dY = bY - aY;

    return sqrtf(dX*dX + dY*dY);

}



bool isRedundant(const string &a, const string &b) {

    if (!startsWith(a, "G1") || !startsWith(b, "G1")) {

        return false;

    }

    

    bool found = true;

    

    float aE = getParameter(a, 'E', &found);

    if (!found) return 0;

    

    float bE = getParameter(b, 'E', &found);

    if (!found) return 0;

    

    if (fabsf(aE - bE) < EXTRUSION_THRESHOLD) {

        // this distance check is probably unnecessary as travel moves are automatically filtered out (they don't have an E parameter)

        float dist = xyDistance(a,b);

        if (dist < 0) return false; // no distance data

        return dist < RESOLUTION_THRESHOLD;

    }

    return false;

}



int main(int argc, const char * argv[]) {

    if (argc < 2 || argc > 3) {

        cout << "usage: gcodefix [input filename] [output filename]\n";

        return 0;

    }


    string filename = argv[1];

    

    cout << "Parsing " << filename.c_str() << ":" << endl;

    

    // output file

    bool output = false;

    ofstream outstream;


    if (argc == 3) {

        output = true;

        string outfile = argv[2];

        outstream.open(outfile);

        if (!outstream.is_open()) {

            cout << "unable to open output file " << outfile.c_str() << "\n";

            return 0;

        }

    }

    

    

    // read file

    int totalCount = 0;

    int duplicateCount = 0;

    

    bool relativeMotion = false;

    string line;

    string previousLine = "";

    ifstream infile(filename);

    if (infile.is_open()) {

        while (getline (infile,line)) {

            totalCount++;

            

            // don't filter relative motion gcode

            if (startsWith(line,"G91")) {

                relativeMotion = true;

            } else if (startsWith(line,"G90")) {

                relativeMotion = false;

            }

            

            // ignore redundant gcode

            if (!relativeMotion && startsWith(line, "G1") && isRedundant(previousLine, line)) {

                duplicateCount++;

//                cout << totalCount << ": " << line << endl;

                continue;

            }

            

            if (output) {

                outstream << line << endl;

            }

            

            previousLine = line;

        }

        

        infile.close();

        if (output) {

            outstream.close();

        }

        

        printf("Finished: %d / %d Lines Removed [%f%%]\n", duplicateCount, totalCount, (duplicateCount * 100.0f)/totalCount);


    } else {

        cout << "Unable to open input file " << filename.c_str() << endl;

    }

    

    return 0;

}


Leon Grossman

unread,
Oct 21, 2015, 9:38:38 AM10/21/15
to Smoothieware Support
Note that when I say "locks up when operating", I mean unresponsive UI while processing not program lock up...

Leon Grossman

unread,
Oct 22, 2015, 1:14:01 AM10/22/15
to Smoothieware Support
It's still slow (230 seconds for a 23MB file) but here is a decent version compile for Mac with a background thread. Note: the app is not signed.




On Wednesday, October 21, 2015 at 7:47:59 AM UTC-5, Leon Grossman wrote:
S3DFix.zip

Richard Chilton

unread,
Oct 22, 2015, 7:18:17 AM10/22/15
to smoothiewa...@googlegroups.com
I had a quick look at your code out of curiosity as I’ve been slowly building my Swift skills up too. Looks good.

You’re probably already aware that buffering the output in fileData string is the most likely cause of the performance issue, but I though I should throw it out there and break the fundamental rule of performance optimisation by optimising before profiling.

Also hasPrefix should just work without needing the string length checks which I can see were inherited from the C++ code you started with.



--
You received this message because you are subscribed to the Google Groups "Smoothieware Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to smoothieware-sup...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<S3DFix.zip>

Leon Grossman

unread,
Oct 22, 2015, 8:35:59 AM10/22/15
to Smoothieware Support, richard...@tracknite.co.uk
Thanks for taking a look!

My original plan was to append to a text file but Swift doesn't have an inherent way to do that with Strings and I would need to learn to work with NSStrings which took a back seat to functioning code. That said... The fileData performance was checked before I inserted the processing functions and the + operator seems to be optimized. The read file in, append to big string, save file out operation was super fast.

My big performance hit appears to be all of the string parsing I'm doing for the get parameter function. Basically, all of the operations I'm doing for the meat of the operation are computationally expensive. My current plan is to eliminate the many get parameter calls for an optimized getParameters that returns X, Y, E as one function.


On Thursday, October 22, 2015 at 6:18:17 AM UTC-5, Richard Chilton wrote:
I had a quick look at your code out of curiosity as I’ve been slowly building my Swift skills up too. Looks good.

You’re probably already aware that buffering the output in fileData string is the most likely cause of the performance issue, but I though I should throw it out there and break the fundamental rule of performance optimisation by optimising before profiling.

Also hasPrefix should just work without needing the string length checks which I can see were inherited from the C++ code you started with.
On 22 Oct 2015, at 06:14, Leon Grossman <leongr...@gmail.com> wrote:

It's still slow (230 seconds for a 23MB file) but here is a decent version compile for Mac with a background thread. Note: the app is not signed.




On Wednesday, October 21, 2015 at 7:47:59 AM UTC-5, Leon Grossman wrote:
This has now reached the point of needing it's own topic. Whether it's to avoid the dreaded crash or to improve your performance of USB printing, these apps are for you.

I wanted to learn swift so I made this app. https://github.com/leonmf/S3DFix


--
You received this message because you are subscribed to the Google Groups "Smoothieware Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to smoothieware-support+unsub...@googlegroups.com.

Leon Grossman

unread,
Oct 22, 2015, 8:46:44 AM10/22/15
to Smoothieware Support, richard...@tracknite.co.uk
I just removed that redundant string length check and... wow. The performance improvement was dramatic.  Not as much as my exuberant commit message indicated but still another 15% improvement.

I'll keep poking at it and then figure out a way to post back from my background thread for a progress bar.


On Thursday, October 22, 2015 at 6:18:17 AM UTC-5, Richard Chilton wrote:

Richard Chilton

unread,
Oct 22, 2015, 9:06:28 AM10/22/15
to smoothiewa...@googlegroups.com
Wow, I am surprised I fully expected appending to fileData a line at time up to 233MB to be the culprit.

Nice work on the +15%.

I wondered if instead of calculating the xydistance then change the function to mean isSignificantlyFarAway() -> Bool and pass in the tolerance and square it at the top of the function. This would avoid caculating the square root of x^2 + y^2. Or even precalculate tolerance square at the top of the app.

Is there a public file somewhere I can use as test data? Maybe commit it to github?



Leon Grossman

unread,
Oct 22, 2015, 9:38:53 AM10/22/15
to Smoothieware Support, richard...@tracknite.co.uk
I'm surprised too. I might not be completely off the hook, though. The 23MB file with no processing only took a couple of seconds to run. However, the 3.3MB file took 13 seconds (4 s/MB) and the 23MB file took 196 seconds (8.4 s/MB) so we could be seeing a performance hit as the memory space grows.

Removing the SQRT function is a great idea! The class property can set the square at the top of the app. I still think my killer is the string functions but I'm betting this gives us nontrivial gains since SQRT is relatively computationally expensive. Unfortunately, I won't be able to touch this again until later tonight.

As for a test file, I don't have access to the files I've been using while here at work but any medium to large gcode file will work for performance testing. I've been running a 3MB file for short tests and a 23MB file for long tests.

To unsubscribe from this group and stop receiving emails from it, send an email to smoothieware-support+unsub...@googlegroups.com.

Triffid Hunter

unread,
Oct 22, 2015, 10:59:54 PM10/22/15
to smoothiewa...@googlegroups.com, richard...@tracknite.co.uk
On 22 October 2015 at 21:38, Leon Grossman <leongr...@gmail.com> wrote:
Removing the SQRT function is a great idea! The class property can set the square at the top of the app. I still think my killer is the string functions but I'm betting this gives us nontrivial gains since SQRT is relatively computationally expensive. Unfortunately, I won't be able to touch this again until later tonight.

Fwiw most modern desktop/laptop processors have a sqrtf and sqrtd instruction in the FPU that should be quite fast, it's smaller chips like smoothieboard and older processors where sqrt is typically computationally expensive.

I have an old post-processing script in my Slic3r directory from before slic3r had simplification, it seems to run at about 2MB/s on my laptop. Check it at http://triffid-hunter.no-ip.info/decimate.pl.txt

Mikk Kiilaspää

unread,
Oct 23, 2015, 6:52:21 AM10/23/15
to Smoothieware Support
Updated my SimplifyS3D page, it now uses about ~70% less time to process large files (tested with a 14MB file).
Another feature is that it allows you to process multiple files at once, just select multiple files at the same time through a single dialog.

Mikk Kiilaspää

unread,
Oct 23, 2015, 9:39:34 AM10/23/15
to Smoothieware Support
Also, drag'n'drop support is now enabled for your ease of use.

Leon Grossman

unread,
Oct 23, 2015, 9:42:06 AM10/23/15
to Smoothieware Support, richard...@tracknite.co.uk
I did a test of this yesterday afternoon in LabVIEW and determined the computational savings to be negligible.

I tried a smarter algorithm for parsing the lines into values and the new was slightly slower. I haven't figured out why yet.

I'm still trying to get file appends working in Swift but I keep getting permissions errors for the same file paths I have no problem with when writing the whole file.

But, still... progress is being made!

Leon Grossman

unread,
Oct 23, 2015, 9:42:58 AM10/23/15
to Smoothieware Support, richard...@tracknite.co.uk
Computational savings of removing SQRT...

Leon Grossman

unread,
Oct 23, 2015, 9:43:41 AM10/23/15
to Smoothieware Support
Awesome. You're making my tool obsolete before it's even finished. :) Keep up the good work!

Leon Grossman

unread,
Oct 24, 2015, 3:39:14 PM10/24/15
to Smoothieware Support
I was getting a bit worried that Mikk's web based code completely obsoleted my code. It is true that his code is still faster than my code but I've improved performance by an order of magnitude just by switching from string operations to working with arrays of bytes. I'm sure there's more performance to be had but I'm happy that even large 20+MB files that used to take 200seconds now take less than 30.

I'm still happy to have more feedback on performance improvements.

Leon
S3DFix 2015-10-24.zip
Reply all
Reply to author
Forward
0 new messages