Adding WiFi to your bot with Toshiba FlashAir SD Card

[Jake]

Over on another thread, there was some discussion on using a WiFi enabled SD card in the Replicator.  (https://groups.google.com/d/topic/makerbot/vycvaKTwue0/discussion) A few options were discussed, including the EyeFi and the Toshiba FlashAir.

I decided to buy a Toshiba FlashAir and see if I could get it to work.  The ability to both download from and upload to the card is listed as a feature, and mentioned in many reviews.  Unfortunately, there is no mention of the upload feature in the documentation or in the card's UI.  Out of the box, the card broadcasts as a WiFi base station.  When you connect to the card's SSID, you then get a web server that allows you to read the files on the card.  That's about it. Settings are configured by a text file stored in a directory on the SD card.

Luckily, it seems like there is a trove of undocumented features on the card, including upload ability, and the ability to join an existing network.

So what's the secret sauce?  Its all in the CONFIG file:

[Vendor]

CIPATH=/DCIM/100__TSB/FA000001.JPG

VERSION=F24A6W3AW1.00.03

CID=02544d53573038470750031e0c00cc01

PRODUCT=FlashAir

VENDOR=TOSHIBA

UPLOAD=1

APPMODE=5

APPNAME=Makerbot

APPSSID=MyNetworkName

APPNETWORKKEY=MyNetworkPassword

LOCK=1

APPAUTOTIME=300000

The key is the undocumented "APPMODE" setting.  APPMODE=4 (default) will make the card broadcast its own network.  APPMODE=5 (undocumented) will have the card connect to your exisitng network.  APPNAME is the host name the card uses when in APPMODE=4, not sure it does anything when APPMODE=5.

The second key is the "UPLOAD=1" line, which enables the upload functionality.

LOCK=1, as far as I can tell, informs the card whether the initial setup has been performed.  APPAUTOTIME is the time before the card shuts down the WiFi (in 1/1000's of a sec).  It seems to me that removing this key completely doesn't leave the card active indefinitely, so I've gotta try putting in a large number there.

Other settings that exist but I haven't really played with: APPSSID, APPCHANNEL, HTTPDMODE, HTTPDUSER, HTTPDPASS, APPMINTIME, APPMAXTIME, DNSMODE, REDIRECT, APPEXT, APPTYPE, AGINGTIME, WLANSTAMODE, DOMAINNAME, APMODE, NOGATEWAYMODE

Uploading is simple,  just go to <SDCARDIP>/upload.cgi.  A very small HTML form will appear where you can pick a file and upload it.  You'll get back "Success" if it worked an "NG" (No Good?) if it didn't.  You can also use curl from the command line

 curl -F file=@FILETOUP.x3g -F submit=submit http://SDCARDIP/upload.cgi

 

There are some limitations.  First, the upload feature only supports 8.3 filenames.  Second, any files uploaded are uppercased.  So "NewArm.x3g" becomes "NEWARM.X3G".  Current versions of bot firmware will only show lowercase extensions, but Dan has kindly decided to tentatively include a case insensitive filter in the next release of Sailfish.  (It takes up some precious bytes, so it may eventually need to be removed someday)

Thirdly, the card doesn't seem to stay connected to the network indefinitely.  The card is designed for cameras, and leaving WiFi on while you're out taking pictures would be a battery drain, so it shuts down in a predefined internal.  (Configurable).  Also, I'm not sure how stable their WiFi/IP stack is.  I'm hopeful that I (or someone else) can figure out how to get it to stay on and connected permanently.

Finally, the card's a little slow, but its not terrible for transferring X3Gs.

I have not tried changing files in use by the bot during a print.  I think that would be a bad idea.

Anyway, there more but I think this post is long enough for now.  Is it worth $50 for one of these cards?  MAYBE-- it has potential if we can get the network link to stay online and reliable.  Even without that, it still might be preferable to swapping out SD cards all the time.

[Jake]

Just a note, since the card is an 8GB SDHC card, you'll need to be running sailfish for this to work at all, in case that wasn't clear.

[RocketGuy]

I'm attempting to get this to work, is the config file viewable somewhere on the card?  How did you get to it?  (I'm Macosx/unix capable, but not windows, so I'm hoping it wasn't a windows utility).  I attempted to find it via the wifi web server, but no dice.  Can you clarify your specific steps?  

Thanks-

-RG

[MadReasonable]

Wow, thanks for your work on this. I had given up on the idea after reading that the card created it's own network. How are you getting around the uppercase issue right now?

[Dan Newman]

On 30 Jul 2013 , at 5:16 PM, MadReasonable wrote:

> Wow, thanks for your work on this. I had given up on the idea after reading that the card created it's own network. How are you getting around the uppercase issue right now?

I'll let Jake answer the other questions but as to the case sensitivity issue,
Sailfish 7.5 is case insensitive as regards .s3g vs. .S3g vs. .s3G vs. .S3G.
Likewise for x3g. And it's never been an issue on ToMs and Cupcakes which
don't care one iota about file extensions. Worrying about the file extension
was something MBI did in the Replicators and Sailfish emulated that behavior.

Dan

[Jake]

I am not near my machine, but I'll answer from memory. The config file is plain text in a SD_WLAN folder if I recall. Mount the SD card in the SD slot on your mac and you should be able to see and edit the config file.

[MadReasonable]

Just ordered the SD card :)

[Jetguy]

Hey, I just wanted to link the threads here because I presented a neat idea if used with the Toshiba card that could provide both file transfer and control.

https://groups.google.com/forum/#!topic/makerbot/osmlB9UiUUE

 

Yes, it requires you mod the mainboard. It's NOT that bad, only 2 lines. It won't work with Makerware but would work with Rep-G.

http://store.3drobotics.com/products/3dr-radio-telemetry-kit-915-mhz

 

This is a USB to serial modem wireless pair of adapters.

Basically it replaces the USB to serial conversion that normally happens on the mainboard.

 

Again, I know this will work for control (I wouldn't print over it) but does require a mod to the mainboard. I can demonstrate this next weekend and document for a mightyboard rev. E and Jake would have to be down to investigating the mod for rev-G and H boards in the 2 series.

[Jake]

I posted s a really simple "use at your own risk", not much error handling, python script to upload files to the FlashAir at the bottom of the blog page.  (http://www.extrud3d.com/flashair)

It uses the "Poster" library which you must install: https://pypi.python.org/pypi/poster/

Usage is simple:

flashair_uploader.py -v -c makerbot.myhomenet.private PRINTME.x3g

the -v will subsequently download the file back off the SD card and verify that it went up properly. The -c parameter is for the IP of the card. It uploads into the root directory of the card.

It uses ANSI code to render the progress bar. Not sure how well it will work on the Windows Command line, but works great MacOS. Maybe someone will improve it using the CURSES library or something? If you make improvements, post them here.

I haven't done much testing with concurrency and how replacing files works. I think its possible for you to upload a file to the card over the WiFi interface, and verify it and the SD card interface still has an old version cached. More testing is necessary. Let me know if you discover anything. I recommend using a new filename each time just to be safe.

Anyway, this is 100% how I upload files to by bot now.  The only time I eject the card now is when I ham-fistedly accidentally press down on it with my wrists while working in the machine.

[Jake]

I'd normally do more research before I just "throw this over the wall" but I'm at AirVenture this week and don't really have the time to fully research this.

The Rev G board has an unpopulated header that exposes PortD,3 and PortD,4 which I believe are the RX/TX pins of the UART1.  As far as I can tell, these pins are mapped to TX_ENABLE and RX_ENABLE in the firmware.  I think these are for RS485 comms, and not used on Mightyboards, but I could be wrong.  Perhaps a special version of the firmware could be made with the main UART0 for UART1.  Then you could connect your wireless solution to the a header rather than making modifications to the mightyboard.

I'm not suggesting that this is a GOOD solution, but its a possible solution.  I'm not sure how this would affect firmware updating, etc.  If the bootloader remains using UART0 I'd think it would still work for USB.  Also the 1280 is space constrained, so having a  UART port toggle might not be possible due to size constraints.  Anyway, just thinking out loud.

Also has anyone seen a REV H board in the wild?  Wonder what's different from Rev G?

[Big-E]

Cool, I'm not too far from Oskosh; About an hour away. Thought about checking AirVenture out this year, but I've been sort of busy. Have fun :)

[Richard]

Wow, nice work!

I want to make sure I understand what you are saying and implying.

So it sounds like you can communicate with the card remotely through your local network to add or remove files but you still have to go to the machine to select a file and start the print, is that correct?

[Dan Newman]

I'll let Jake answer for what he's doing specifically. I just want to mention
that with, for instance, MBI's s3g.py (used by Conveyor) you could then have
a very simple python script which would then tell the bot over USB which file
to print from the SD card. (How do Cupcake users tell a bot which file to print
off of SD card? Cupcakes do not have LCD displays. A good guess is that there's
a command which tells the bot which SD card file to print.)

Mind you, if you have the latest Sailfish then you need a one line fix to
s3g.py which I can provide.

Dan

[Jake]

Yes. I currently load the files to the SD over wifi and then start the print from the menus.

[RocketGuy]

Thank you Sir!  

I will attempt that this evening.  I just extended my network out to the workshop, so it'd be really cool to actually use it for something besides wasting time on facebook when I should be working on something.

[Paige Warner]

I'd love to know what that one line fix to s3g.py would be.  I just switched over to sailfish
and would like to start transferring my files without swapping cards.  Mind you, my bot is
only a few feet away from my mac but I still don't like swapping cards and pushing more
buttons than I have too...

[Dan Newman]

On 31 Jul 2013 , at 2:17 PM, Paige Warner wrote:

> I'd love to know what that one line fix to s3g.py would be. I just
> switched over to sailfish
> and would like to start transferring my files without swapping cards. Mind
> you, my bot is
> only a few feet away from my mac but I still don't like swapping cards and
> pushing more

> buttons than I have too…

Find StreamWriter.py (streamWrite.py in older versions) and find the section
around line 116 which reads

except makerbot_driver.RetryableError as e:
# Sent a packet to the host, but got a malformed response or timed out waiting
# for a reply. Retry immediately.

self._log.debug('{"event":"transmission_problem", "exception":"%s", "message":"%s", "retry_count"=%i}', type(e), e.__str__(), retry_count)

self.total_retries += 1
retry_count += 1
received_errors.append(e.__class__.__name__)


and add a flush call so that it now reads

except makerbot_driver.RetryableError as e:
# Sent a packet to the host, but got a malformed response or timed out waiting
# for a reply. Retry immediately.

self._log.debug('{"event":"transmission_problem", "exception":"%s", "message":"%s", "retry_count"=%i}', type(e), e.__str__(), retry_count)

# We're going to go back and try again -- flush the input buffer
self.file.flushInput()

self.total_retries += 1
retry_count += 1
received_errors.append(e.__class__.__name__)

That solves the underlying problem. Conveyor sends command X and incorrectly assumes that
the response it gets back is to command X. If it gets an error response, it resends command X.
Problem is that the response may have been to line noise or another process sending data down
the wire after which there will then be a response to command X. So then Conveyor gets two
responses to the two command X's which it sent. Net, Net, Conveyor gets confused. The s3g
protocol simply doesn't adequately take comms errors into account and gets all out of sync
and confused when they happen. When the bots didn't send any error responses back, Conveyor
didn't get confused this way. However, when you don't send error responses, then Conveyor can
make other incorrect assumptions. By flushing the I/O channel, the first response to command X
is wiped and Conveyor doesn't get confused when it resends command X.

NOW, you may not be out of the woods when you make this change. You may still have
issues with Conveyor (s3g.py and support code) seeing error responses for which MBI has
not added code to handle and thus they appear as an "unknown" error rather than nicely
categorized where they belong. But for just sending a "print this file", you may/should
be okay.

Dan

[Dan Newman]

TYPOS

> Find StreamWriter.py (streamWrite.py in older versions) and find the section

"streamWriter.py" in older versions.

> except makerbot_driver.RetryableError as e:
> # Sent a packet to the host, but got a malformed response or timed out waiting
> # for a reply. Retry immediately.
>
> self._log.debug('{"event":"transmission_problem", "exception":"%s", "message":"%s", "retry_count"=%i}', type(e), e.__str__(), retry_count)
>
> # We're going to go back and try again -- flush the input buffer
> self.file.flushInput()

Be sure to get that self.file.flushInput() indented correctly…. I didn't when I
cut-n-pasted things.

Dan

[Paige Warner]

When did an email ever format python correctly?..

So it took me a little bit to familiarize myself with getting a python env setup
for s3g since I haven't dug into ever before. But after looking at the conveyor
code a bit and finding an entry point to a valid s3g object though MachineFactory
I was able to do this:

>>> import makerbot_driver
>>> mf = makerbot_driver.MachineFactory('/Library/MakerBot/s3g/profiles')
>>> retObj = mf.build_from_port('/dev/tty.usbmodemfa1421')
>>> retObj.s3g.playback_capture('Stretchlet.x3g')

4 lines, and now my Rep2 is printing me an orange stretchlet, I think I'll wear it tomorrow… :)

This is awesome, thanks Dan!!!
~paige

> --
> You received this message because you are subscribed to a topic in the Google Groups "MakerBot Operators" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/makerbot/Rst5ZIfs5L0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to makerbot+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Paige Warner]

And here's a basic routine to get the filenames on the sd card through s3g as well:

(this is all just me tinkering in the interpreter so forgive the short names)

…continued from previous example…

s = retObj.s3g

files = []

# Get the first item.
filename = str(s.get_next_filename(reset=True))[:-1]

# Items returned are a bytearray with a trailing null (\x00) so I'm just slicing it off,
# and checking for whats left. When you hit the end of the list, the filename will
# be an empty string, which will kick us out of the while loop.
while filename:
files.append(filename)
filename = str(s.get_next_filename(False))[:-1]

print files
['Stretchlet.x3g', 'Mr Jaws.x3g', 'Comb.x3g', 'Nut and Bolt.x3g', 'Chain Links.x3g', ...]

My FlashAir is on it's way, and then my 3 foot commute from my computer to my bot will finally be eliminated! :)

~paige

[RocketGuy]

Ah, had to go in via terminal and use ls -al , it's invisible in finder.  

Now to edit it with vi...

On Tuesday, July 30, 2013 6:07:53 PM UTC-7, Jake wrote:

[Dan Newman]

On 1 Aug 2013 , at 8:12 AM, RocketGuy wrote:

> Ah, had to go in via terminal and use ls -al , it's invisible in finder.

You should be able to enable the display of "hidden files". (I run with
that all the time in finder.)

Dan

[Dan Newman]

On 1 Aug 2013 , at 12:42 AM, Paige Warner wrote:

> When did an email ever format python correctly?..
>
> So it took me a little bit to familiarize myself with getting a python env setup
> for s3g since I haven't dug into ever before. But after looking at the conveyor
> code a bit and finding an entry point to a valid s3g object though MachineFactory
> I was able to do this:
>
>>>> import makerbot_driver
>>>> mf = makerbot_driver.MachineFactory('/Library/MakerBot/s3g/profiles')
>>>> retObj = mf.build_from_port('/dev/tty.usbmodemfa1421')
>>>> retObj.s3g.playback_capture('Stretchlet.x3g')
>
> 4 lines, and now my Rep2 is printing me an orange stretchlet, I think I'll wear it tomorrow… :)
>
> This is awesome, thanks Dan!!!

Not a problem, and thanks for posting that. I almost wrote previously that it was
a simple few lines of Python but decided not too. (What might be simple for one
person may not be for another.)

Dan

[David Clunie]

I've got a eye-fi card 8gb.  I'll have to try this with my replicator2x.  Would save alot of going back and forth and allow for me to not have a dedicated netbook hooked up to the machine or running the sd card back and forth between my main machine where i do all the modeling in cinema 4d then transporting it to the sd card after a quick slice and dice in makerware....

-dclunie

On Saturday, June 1, 2013 6:48:55 AM UTC-7, Jake wrote:

[Dan Newman]

On 1 Aug 2013 , at 6:30 PM, David Clunie wrote:

> I've got a eye-fi card 8gb.

eye-fi cards are one way: you can read from them over wi-fi but cannot
write to them over wi-fi.

Dan

[David Clunie]

you know I didn't even think about that, but yah now i look back I was only ever pulling info down and now ever deleting or writing new info up to the card while it was in camera.  (I used to have it using a 3rd party wifi spot connect to my netbook and auto pull down the images for an "unlimited" drive space while i was out an about.  had the netbook in a back pack.)  

Too bad there isn't a way to tweek the eye-fi cards, that would be ideal for me since i don't really use the eye-fi anymore, its just faster to pull out the cards and transfer via built in sd slot on my laptop/netbook/desktop.

[Jetguy]

Just got my Flash Air today and rocking the wireless transfers!!!!

Thanks for the work on this and sharing.

[Andy Chen]

I got my Flash Air working today. It's awesome. Much thanks for getting this all figured out.

I made a small change to Jake's script to use a default server. Now I can do this, and I'm ready to print:

> sendx3g model.x3g

model.x3g [########################################] 100.00% DONE

Whoohoo!

[MadReasonable]

I also got this working pretty easily thanks to Jake's work with the config file.  I've set APPAUTOTIME to 2147483647, or about 25 days.  The card might support max unsigned int32 or higher but this works fine for me.

Does anyone else have problems with special characters like '[', ']', or '-' ?

[Dan Newman]

On 11 Aug 2013 , at 12:02 PM, MadReasonable wrote:

> I also got this working pretty easily thanks to Jake's work with the config
> file. I've set APPAUTOTIME to 2147483647, or about 25 days. The card
> might support max unsigned int32 or higher but this works fine for me.
>

> Does anyone else have problems with special characters like '*[*', '*]*',
> or '*-*' ?

I believe that the Toshiba FlashAir doesn't implement the LFN extension.
Likely the Toshiba engineers didn't want to handle UTF-16 *and* didn't want
to omit that support -- didn't want to implement a crippled (non-spec) LFN
extension. (Mind you, lots of people implement incomplete forms of LFN,
especially as regards incomplete or non-existent UTF-16 support.) And since
LFN doesn't appear to be implemented, it looks like they went with 8.3 uppercase.
And when you go with 8.3 uppercase, you are supposed to not allow any of the
US-ASCII characters

" * + , / : ; < = > ? \ [ ] |

Dan

[MadReasonable]

http://haxit.blogspot.ca/2013/08/hacking-transcend-wifi-sd-cards.html
I only just saw this and haven't looked into it much, but the Transcend cards might give us a lot more capability.  This guy is connecting with a terminal and running busybox off of his.

[Jetguy]

I reading it now but the first obvious thing was:

The third challenge is that the script kcard_upload.pl only allows PNG, JPG, BMP, and GIF files to be uploaded.

 

Now, he does work around it but it seems to me, 10 times harder to use and hack than the Toshiba.

In other words, in the Toshiba, I add 1 line to the config to allow uploads, and it's easy to access with no hacks required.

 

All I'm saying is that I went through this exact debate of what card to buy as there are a bunch out there. Googling wireless SD card upload is just a bunch of peopel posting about downloading photos from the camera, the intended purpose.

 

So it's good to know that maybe in a pinch this card can work, but I say hands down if you just want to use it and are not  hacker, get the Toshiba .

If you had both in a shopping card and the Toshiba was $20 more, I STILL would buy it.

[MadReasonable]

I agree that the Toshiba FlashAir can be made to work very quickly and wouldn't recommend the Transcend card to anyone who wants something that just works.  I might pick up the Transcend to work on as a hobby,  Once all the pieces are in place it might be just as easy as the FlashAir but with fewer limitations.