Demo - Chrome AI + Mapbox for digging into images

[Raymond Camden]

Ok, technically this is more Mapbox than Chrome AI, but I took my existing "select an image and use Chrome AI to tag the contents" and added the ability to read EXIF, get location, call Mapbox and reverse geocode it. So in the best of worlds, you get:

Where the image was taken

A list of tags for items found in the image

I'll include a screen shot but the last few times I have, it's failed to render, so here goes nothing (just went with an attachment:

Here's the CodePen, https://codepen.io/cfjedimaster/pen/NPGrQaM, and Thomas, forgive me for still not supporting the downloading case. ;)

[Thomas Steiner]

Nice demo, thanks for sharing!

I wonder if you could feed the location data from EXIF into the prompt, so the Prompt API has additional details. Maybe something like this:

<user>Identify objects found in the image and return an array of tags. For additional context, this image was taken in Knokke-Heist, West Flanders, Belgium <image> Remember to respond in JSON that follows this "JSON Schema" specification: {"type":"object","required":["tags"],"additionalProperties":false,"properties":{"tags":{"description":"Items found in the image","type":"array","items":{"type":"string"}}}}<end><model>

It might be more prone to hallucinate things that aren't actually in the image, but I'll let you experiment a bit :-)

Cheers,

Tom

--
You received this message because you are subscribed to the Google Groups "Chrome Built-in AI Early Preview Program Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chrome-ai-dev-previe...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ai-dev-preview-discuss/0b02e772-c13b-4eac-8b97-118ced9f39f1n%40chromium.org.

--

Thomas Steiner, PhD—Developer Relations Engineer (blog.tomayac.com, toot.cafe/@tomayac)

Google Spain, S.L.U.

Torre Picasso, Pl. Pablo Ruiz Picasso, 1, Tetuán, 28020 Madrid, Spain

CIF: B63272603

Inscrita en el Registro Mercantil de Madrid, sección 8, Hoja M­-435397 Tomo 24227 Folio 25

----- BEGIN PGP SIGNATURE -----

Version: GnuPG v2.4.8 (GNU/Linux)

iFy0uwAntT0bE3xtRa5AfeCheCkthAtTh3reSabiGbl0ck

0fjumBl3DCharaCTersAttH3b0ttom.xKcd.cOm/1181.

----- END PGP SIGNATURE -----

[Raymond Camden]

Dang you - that's a good idea. So I tried a few things.

One - just adding to the prompt:

Use this photos location to help determine what is in the photo: ${locationStr}

This wasn't terribly helpful - it did return the location in the list of tags (Hungary, etc), but it was mainly repeating what the location result found. 

You can see that here: https://codepen.io/cfjedimaster/pen/QwjKjVg

I then forked this again, got rid of the tag idea, and just said:

 prompt = `What building is shown in this picture, found at: ${locationStr}`;

And wow... it seemed to work *dang* good. This demo is brittle as heck, but can be found here: https://codepen.io/cfjedimaster/pen/yyYaYRv

--

===========================================================================
Raymond Camden

Email : raymon...@gmail.com
Blog : www.raymondcamden.com
Twitter: raymondcamden

[Thomas Steiner]

Thanks for sharing your continued experimentation. Let us know once you played with more images. I'd be interested in hearing about your results with taking benign pictures (for example of a bird) at famous locations (for example of the location of the White House) to see if this causes the Gemini Nano model in Chrome to hallucinate based on this priming. Also interested in seeing how Phi-4 in Edge does in comparison. Sorry, I'm just boarding a plane, so jotting down my thoughts on my phone. Don't feel obliged to actually run these experiments of course. 

Thomas Steiner, PhD—Developer Relations Engineer (blog.tomayac.com, toot.cafe/@tomayac)

Google Spain, S.L.U.

Torre Picasso, Pl. Pablo Ruiz Picasso, 1, Tetuán, 28020 Madrid, Spain

CIF: B63272603

Inscrita en el Registro Mercantil de Madrid, sección 8, Hoja M­-435397 Tomo 24227 Folio 25

----- BEGIN PGP SIGNATURE -----

Version: GnuPG v2.4.8 (GNU/Linux)

iFy0uwAntT0bE3xtRa5AfeCheCkthAtTh3reSabiGbl0ck

0fjumBl3DCharaCTersAttH3b0ttom.xKcd.cOm/1181.

----- END PGP SIGNATURE -----

[Raymond Camden]

FYI, I cleaned up the demo a bit today so it can show the AI results *without* the location info and *with* so you can compare and contrast. Still the same url, https://codepen.io/cfjedimaster/pen/QwjKjVg?editors=1010

But as an example of one of my pics from Austria:

Tags without location: castle, hill, sky, clouds, trees, buildings, architecture, Austria, Europe, historic, religious building, dome, cross

Found location: Salzburg, Salzburg, Austria
Tags with location: Hohensalzburg Fortress, Salzburg, Austria, castle, hill, sky, clouds, buildings, architecture, historic, religious building, dome, cross, fortress, mountain, landscape

Going to blog this later today.

[Clark Duvall]

This is really cool thanks for sharing Raymond! One recommendation: I would remove the {type:'audio'} from your availability check to avoid the demo being disabled for devices that can't run the audio model (but can still handle image input).

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ai-dev-preview-discuss/c2adc6fb-1205-4dff-9a0f-f8a748441a4fn%40chromium.org.

[Raymond Camden]

Ah thanks - I had removed it from the create, forgot to do so on availability.