Shaders in pi3D?
187 views
Skip to first unread message
Louise Lessél
Jul 18, 2020, 11:21:31 AM7/18/20
to pi...@googlegroups.com
Hi,
I have a question about the shader implementation on pi3D and specifically porting Shadertoy shaders and using uniforms. I am trying to make shaders more approachable for beginning programmers starting out. I did this for p5js (https://itp-xstory.github.io/p5js-shaders/#/). Now I want to try on the Pi, and came across pi3D. Exciting work everyone!
I was looking at the shader implementation, but I am not a hard core low level graphics card programmer,,,, so there's some stuff I don't understand.
I am trying to include the entire shadertoy boilerplate code ( iResolution, iTime etc ) in the beginning of the code.
But I came across an issue when using uniforms and #defines in the code I was porting.
In one of the shader examples I am trying to port (http://www.shadertoy.com/view/XsXXDnv) the creator #defines the iResolution as r and uses that. This doesn't work?
And furthermore, I am not sure how to properly pass in the uniforms for iResolution etc.
----
I am able to pass in uniforms doing something like in my .py file:
while DISPLAY.loop_running(): # count time as iterations instead t += 1* 0.01 #print(t) # set uniforms, for GL reasons, these are set on '48'? sprite.set_custom_data(48, [t, WIDTH, HEIGHT]) # draw the shader on the sprite (flat rectangle) sprite.draw()
But here is my issue from the fragment shader (read the comments):
| //#version 330 | |
| #ifdef GL_ES | |
| precision mediump float; | |
| #endif | |
| // Add this in so I can pass in uniforms | |
| uniform vec3 unif[20]; | |
| // is a vec3 so there is three spaces to place a uniform // I assume I can make more uniforms by making this a vec5? Or how? | |
| // for some reason they are in array place index 16 | |
| // 16,0 is uniform0 | |
| // 16,1 is uniform1 | |
| // 16,2 is uniform2 | |
| // ShaderToy variables boilerplate | |
| uniform vec3 iResolution; // viewport resolution (in pixels) | |
| uniform float iTime; // shader playback time (in seconds) | |
| uniform float iTimeDelta; // render time (in seconds) | |
| uniform int iFrame; // shader playback frame | |
| uniform float iChannelTime[4]; // channel playback time (in seconds) | |
| uniform vec3 iChannelResolution[4]; // channel resolution (in pixels) | |
| uniform vec4 iMouse; // mouse pixel coords. xy: current (if MLB down), zw: click | |
| uniform sampler2D iChannel0; // input channel | |
| uniform sampler2D iChannel1; | |
| uniform sampler2D iChannel2; | |
| uniform sampler2D iChannel3; | |
| uniform vec4 iDate; // (year, month, day, time in seconds) | |
| // Shader from shadertoy | |
| // http://www.pouet.net/prod.php?which=57245 | |
| // If you intend to reuse this shader, please add credits to 'Danilo Guanabara' | |
| // http://www.shadertoy.com/view/XsXXDnv | |
| // Why don't #define work? If I comment them in, the code breaks | |
| //#define t iTime | |
| //#define r iResolution.xy | |
| void main() { | |
| // The #defines don't work so instead I have been doing this. | |
| // I found that I HAVE TO assign the unif[...] in the main loop? | |
| float t = unif[16][0]; | |
| vec2 r = vec2(unif[16][1],unif[16][2]); | |
| // Additionally, actually using the uniforms from the | |
| // shadertoy boilerplate doesnt work, and I can't assign them here? | |
| // Commenting this in will break the code. | |
| //iResolution = vec3(unif[16][1],unif[16][2], 0.0); // dont work | |
| //iResolution.xy = vec2(unif[16][1],unif[16][2]); // dont work | |
| // From the original shader code | |
| vec3 c; | |
| float l,z =t; | |
| for(int i=0;i<3;i++) { | |
| vec2 uv,p=gl_FragCoord.xy/r; | |
| uv=p; | |
| p-= 0.5; | |
| p.x*=r.x/r.y; | |
| z+=.07; | |
| l=length(p); | |
| uv+=p/l*(sin(z)+1.)*abs(sin(l*9.-z*2.)); | |
| c[i]=.01/length(abs(mod(uv,1.)-.5)); | |
| } | |
| gl_FragColor=vec4(c/l,t); | |
| } |
_________________
Py program here:
Fragment and Vertex here:
---
Paddy
Jul 18, 2020, 2:16:58 PM7/18/20
to pi3d
Louise, it's great that you're having a go at this. I did do a couple of shadertoy experiments with pi3d but mainly took the bits of the code that did the shading and shoe-horned them into the existing pi3d structure, cutting bits out where there was something I didn't think I needed.
However it would be nice to build a standard framework so people can slot in a shader without too much hacking.
In the early days of pi3d we tried various ways to get the frame rate up, and at one point, sent info to the vertex shader to allow it to do the transformation matrix multiplication. Although that worked after a fashion I found that numpy could do the linear algebra pretty efficiently and eventually I managed to eliminate the slow steps switching between python, ctypes and numpy arrays. TLDR; There is a block of 60 float values in each pi3d.Shape which is passed to the shaders as an array of 20 vec3. Although some are used by different bits of pi3d camera, light and other shaders. I think for shadertoy demos they could all be used but a) there aren't any uniform int passed so iFrame might need to be cast. You can't write to uniform variables which is why your iResolution doesn't work, you would have to do something like:
vec3 iResolution = vec3(unif[16], 0.0);
but it has to be in the main() function I think.
I will have a look at the shader you reference and see what I come up with.
Paddy
Louise Lessél
Jul 18, 2020, 5:45:14 PM7/18/20
to pi3d
Thank you very much Paddy!
Excited to see what you come up with.
Yes your experiments was actually how I found pi3d! But I couldn't find the source code for your program so I couldn't take it apart :D --- best to reach out then.
I've been trying my hand at this for about a week now, and the only way I have been able to get it to function is to insert in the main() function. I just worry that when shaders become increasingly complex (like the one slightly more involved one I have linked below), that putting everything in the main() function will leave it inaccessible to other functions in the code. Like in case a function outside of main() was to use the iResolution or iTime... though I haven't come across that in any of the ones I am looking at currently.
I am writing a python implementation in Touchdesigner, that converts the Shadertoy code automatically to be run in pi3D.
(Does all the proper replacements).
Paddy
Jul 18, 2020, 6:19:21 PM7/18/20
to pi3d
Louise, I think you should be able to define variables outside main() just not set them to uniform variables there. So something like:
vec3 iResolution = vec3(0.0);
void main(void) {
iResolution = vec3(unif[16].yz, 0.0); //yes, .yz missed off before!
Of course it would be possible to define uniform variables with the same names and types used in shadertoy but it's a bit messy (see https://github.com/tipam/pi3d/blob/master/pi3d/Shader.py#L124 and https://github.com/tipam/pi3d/blob/master/pi3d/Buffer.py#L280 it would involve some hacking in the internals of pi3d.
Making a converter sounds like the best bet. There is an #include system in the pi3d shaders (that I added as so many had duplicate code) It basically looks in the pi3d/pi3d/shaders folder or the folder that the shader is in for a file with the name following #include and pastes the text in place (recursively). So you could probably pull boilerplate in that way. There is also the issue of GL/GLES 2/3 which pi3d attempts to determine and convert GLSL code to match.
I might not get a chance to look at this until next week, but will let you know further thoughts.
Paddy
Louise Lessél
Jul 18, 2020, 7:40:52 PM7/18/20
to pi3d
Hi Paddy
Thank you so much! Huge help!
I tested something like this below and it works.
I am curious about the best way to go about setting up all of the uniforms though, since I'm not sure how to use this part properly?
uniform vec3 unif[20];
// is a vec3 so there is three spaces to place a uniform
// for some reason they are in array place index 16
// 16,0 is uniform0
// 16,1 is uniform1
// 16,2 is uniform2
And this stuff: vec3(unif[16].yz, 0.0);
yz?
What if I wanted to pass in many many uniforms? Is there a limit? And how?
------- NEW SHADER CODE TEST THAT WORKS: ----------
//#version 330
#ifdef GL_ES
precision mediump float;
#endif
uniform vec3 unif[20];
// is a vec3 so there is three spaces to place a uniform
// for some reason they are in array place index 16
// 16,0 is uniform0
// 16,1 is uniform1
// 16,2 is uniform2
//---------------------------------------- None of these methods work
//uniform float iTime;
//uniform float iTime = unif[16][0];
//#define t iTime;
//float t = iTime;
//#define t unif[16][0];
//uniform float t;
//uniform float iTime => unif[16][0];
//------------------------------------------ This works
float t = unif[16][0];
float iTime = unif[16][0];
//------------------------
// ShaderToy variables
// I can't include as uniforms... But can recreate as
vec3 iResolution;
/*
uniform vec3 iResolution; // viewport resolution (in pixels)
//uniform float iTime; // shader playback time (in seconds)
uniform float iTimeDelta; // render time (in seconds)
uniform int iFrame; // shader playback frame
uniform float iChannelTime[4]; // channel playback time (in seconds)
uniform vec3 iChannelResolution[4]; // channel resolution (in pixels)
uniform vec4 iMouse; // mouse pixel coords. xy: current (if MLB down), zw: click
uniform sampler2D iChannel0; // input channel
uniform sampler2D iChannel1;
uniform sampler2D iChannel2;
uniform sampler2D iChannel3;
uniform vec4 iDate; // (year, month, day, time in seconds)
*/
// If you intend to reuse this shader, please add credits to 'Danilo Guanabara'
//#define t iTime
//#define r iResolution.xy
void main() {
// I can then do this (set up automatic script to detect ang generate.
// float t = iTime;
//float t = unif[16][0]; // so they CAN be assigned outside the main() and it doesn't have to be here.
vec2 r;
r.x = unif[16][1];
r.y = unif[16][2];
vec3 c;
float l,z =t;
for(int i=0;i<3;i++) {
vec2 uv,p=gl_FragCoord.xy/r;
uv=p;
p-= 0.5;
p.x*=r.x/r.y;
z+=.07;
l=length(p);
uv+=p/l*(sin(z)+1.)*abs(sin(l*9.-z*2.));
c[i]=.01/length(abs(mod(uv,1.)-.5));
}
gl_FragColor=vec4(c/l,t);
}
Paddy
Jul 19, 2020, 5:27:19 AM7/19/20
to pi3d
Louise, In python Shape.unif is defined as a ctypes array of float[60] https://github.com/tipam/pi3d/blob/master/pi3d/Shape.py#L41 although the first few are used for pi3d matrices etc that shouldn't matter in the shadertoy examples as they use their own uniforms. So in python you can write directly to unif (the set_custom_data() is just a wrapper for this)
sprite.unif[0:3] = [WITDH, HEIGHT, 0.0] # iResolution
sprite.unif[3:6] = [0.0, 0.0, 0.0] # iTime, iTimeDelta, iFramethen in the fragment shader do
#define iResolution unif[0]
#define r iResolution.xy //compiles but not checked if it actually works
#define iTime unif[1][0]
#define iTimeDelta unif[1][1]
#define iFrame unif[1][2]So you can do the #defines outside main but for the vec4 and int uniforms I think you will have to do as described before - define the variable outside but the value inisde. Obviously it makes sense to keep the vec3 uniform values at locations in the ctypes code 0,3,6,9 etc
In GLSL it is more efficient to do vector operations where you can (as the GPU is designed for paralel processing of this kind of thing. The .xy notation is a way of extracting or setting part of a vec as a vec. You can use .xyzw .rgba .stpq .uv (probably others) so, for instance
gl_FragColor.rgb *= 0.5;
or
gl_FragColor.rgb = vec3(dot(gl_FragColor.rgb, vec3(0.21, 0.72, 0.07)))and it can do broadcasting as in the first example above or where I did vec3(unif[16].xy, 0.0)
I'm doing this from memory so some might be a bit wrong - I will try some code next week!
Paddy
Paddy
Jul 19, 2020, 5:55:17 AM7/19/20
to pi3d
PS I get a segmentation fault if I try
...
float tmp = unif[18][0]; //but #define is fine, setting value in main()
...
void main(void) {
...but in your code you say it works which seems strange, wonder if it's a GLSL version thing?
Paddy
Jul 19, 2020, 6:22:26 AM7/19/20
to pi3d
PPS it has just occurred to me that as you won't be using the transformation matrix you could pass in variables (or, for that matter use the vertex normals and uv texture coordinates) so it should be possible to pass uniforms in that way. modelviewmatrix is an array of three mat4 values held on the python side in Shape.M which is a numpy array (3, 4, 4) the first two might get some scrambling by the Shape.draw() method (though I think M[0,:3,:] should remain intact) but M[2] is only used in shadow casting so should be available for use so you could put iMouse and iDate in there
sprite.M[2,0,:] = [mousex, mousey, mouseclickx, mouseclicky] # or whatever
...
#define iMouse modelviewmatrix[2][0]
#define iDate modelviewmatrix[2][1]Paddy
Jul 19, 2020, 7:16:28 AM7/19/20
to pi3d
Also, there might be a couple of places like this https://github.com/tipam/pi3d/blob/master/pi3d/Shape.py#L202 where Shape.unif is modified in the first draw() call but then a flag is set to stop code re-running unnecessarily. So you might have to set some of your uniform values *after* the first call of the Display.loop_running(). Or make sure that the blocks of unif that might get initialised by draw() after you had initialised them are ones you re-write every frame.
Louise Lessél
Jul 21, 2020, 6:24:46 PM7/21/20
to pi3d
Thank you Paddy - this was precisely the information I couldn't find ---->
"Louise, In python Shape.unif is defined as a ctypes array of float[60] https://github.com/tipam/pi3d/blob/master/pi3d/Shape.py#L41 although the first few are used for pi3d matrices etc that shouldn't matter in the shadertoy examples as they use their own uniforms. So in python you can write directly to unif (the set_custom_data() is just a wrapper for this)"
:D
So what I get from the documentation, is that I only have 4 * 3 floats (12 floats) to pass in as uniforms? :
16 custom data space 48 50
17 custom data space 51 53
18 custom data space 54 56
19 custom data space 57 59
So in the .py file I do...
sprite.set_custom_data(48, [t, WIDTH, HEIGHT]) and it is really taking up 48, 49, 50?
And in the .fs file it can be fetched as vec3(unif[16][0],unif[16][1],unif[16][2]); ?
Also, what does this line do/mean? Is that the amount of data indexes (0-19)? And if so, can I make it larger like unif[40]?
uniform vec3 unif[20];
Hmm. I am still unable to use #define like you describe and actually get it to run on the pi4.
But I have finished the first version of my Touchdesigner conversion tool that does work and lets you convert from ShaderToy to TouchDesigner to Pi4 in pi3d.
It is still a work in progress but it works now by changing all the #define's to their respective types (and commenting out the original).
So basically the Shadertoy code...
___________________________________________________________________________
// If you intend to reuse this shader, please add credits to 'Danilo Guanabara'
#define t iTime
#define r iResolution.xy
void main(){
vec3 c;
float l,z=t;
for(int i=0;i<3;i++) {
vec2 uv,p=fragCoord.xy/r;
uv=p;
p-=.5;
p.x*=r.x/r.y;
z+=.07;
l=length(p);
uv+=p/l*(sin(z)+1.)*abs(sin(l*9.-z*2.));
c[i]=.01/length(abs(mod(uv,1.)-.5));
}
fragColor=vec4(c/l,t);
}
___________________________________________________________________________becomes .... touchdesigner code (not gonna copy paste that)...
but then becomes pi4 code copy pasted below ... (as you can see I'm still working on the assignment of more uniforms.)
___________________________________________________________________________
// Pi variables
//#version 330 // currently defining the version (eg 330 or 120) breaks the code for some reason
#ifdef GL_ES
precision mediump float;
#endif
uniform vec3 unif[20];
// ShaderToy variables (under implementation as uniforms)
vec3 iResolution = vec3(unif[16][1],unif[16][2], 0.0); // viewport resolution (in pixels)
float iTime = unif[16][0]; // shader playback time (in seconds)
float iTimeDelta = 1.0; // render time (in seconds)
int iFrame = int(1); // shader playback frame
//float iChannelTime[4] = 1.0; // channel playback time (in seconds)
//vec3 iChannelResolution[4]; // channel resolution (in pixels)
vec4 iMouse = vec4(0.5, 0.5, 0.0, 0.0); // mouse pixel coords. xy: current (if MLB down), zw: click
//sampler2D iChannel0; // input channel
//sampler2D iChannel1;
//sampler2D iChannel2;
//sampler2D iChannel3;
//vec4 iDate; // (year, month, day, time in seconds)
// If you intend to reuse this shader, please add credits to 'Danilo Guanabara'
//#define t iTime
//#define r iResolution.xy
float t = iTime;
vec2 r = iResolution.xy;
void main(){
vec3 c;
float l,z=t;
for(int i=0;i<3;i++) {
vec2 uv,p=gl_FragCoord.xy/r;
uv=p;
p-=.5;
p.x*=r.x/r.y;
z+=.07;
l=length(p);
uv+=p/l*(sin(z)+1.)*abs(sin(l*9.-z*2.));
c[i]=.01/length(abs(mod(uv,1.)-.5));
}
gl_FragColor=vec4(c/l,t);
}
Paddy
Jul 22, 2020, 6:30:32 AM7/22/20
to pi3d
Hi, harder than I thought to check stuff out and communicate as no WiFi or phone signal where we're staying!
As you're using frag coordinates you don't need any of the matrix or other unit values so they're all potentially available to use. The vertex shader doesn't need to do matrix multiplication to get to screen coordinates if these are set to 0,0 to 1,1 in pi3d.
Not sure about RPi4 don't have one here to check.
Paddy
Jul 24, 2020, 5:37:23 PM7/24/20
to pi3d
Louise, replied earlier but can't see any trace so maybe I never
pressed send! I will try again:
Back now with internet and RPis to test on. I still can't
get your system to work where you set a variable to a uniform variable
outside the main function. However the following code seems to work OK
when I test it on ubuntu GLES3 and RPi4 GLES3 and RPi3 GLES2. If this
system doesn't work on your setup let me know any details.
Vertex
shader doesn't need to do anything if you set the vertex values to fill
the screen. i.e. a single triangle (-1,-1) (-1,3) (3,-1) You can then
use all the uniform variables for shadertoy that's enough for 123 floats (Shape.unif -> 20 x vec3, Shape.M -> 3 x mat4, Buffer.unib -> 5 x vec3)
import time
import demo
import pi3d
display = pi3d.Display.create(w=800, h=600)
print(display.opengl.gl_id)
sprite = pi3d.Triangle(corners=((-1.0, -1.0),(-1.0, 3.0),(3.0, -1.0)))
shader = pi3d.Shader('shaders/shadertoy01')
sprite.set_shader(shader)
kbd = pi3d.Keyboard()
tm0 = time.time()
while display.loop_running():
sprite.draw()
sprite.unif[3] = time.time() - tm0
sprite.unif[0:2] = [800, 600]
break
shaders/shadertoy01.vs
#include std_head_vs.inc
void main(void) {
gl_Position = vec4(vertex, 1.0);
}shaders/shadertoy01.fs
#include std_head_fs.inc
#define iTime unif[1][0]
#define iResolution unif[0]
void main(void) {
float t = iTime;
vec2 r = iResolution.xy;
vec3 c;
float len, z=t;
for(int i=0; i<3; i++) {
p.x *= r.x / r.y;
}
gl_FragColor = vec4(c / len, t);
}
It would make sense to match the shadertoy uniform variables up with Shape.unif at the beginning. You don't need to start at vec3 16 as you're not using any of the ones prior to that. The vec4 values could be put into a line of the mat4[3] which is Shape.M
Louise Lessél
Jul 26, 2020, 10:18:55 PM7/26/20
to Paddy, pi3d
Thank you Paddy.
I hope you had a lovely trip?
I will try this tomorrow!
I'm wondering if there is a way to control the resolution of a shader, so I can lower the resolution of the shader?
Since I am trying to set the shader resolution (based off of the shadertoy shaders) iResolution.xy to something really low.
But still make it take up the full screen?
I can do this in Touchdesigner no problem. Make the shader 100 x 100 px and make Touchdesigner output fullscreen.
= suuuper pixelated (but very high performance) shader. Nice! Attached two images of what I'm trying to achieve.
But I am having some real issues replicating that on the pi. Where, as you can imagine it'd be pretty important to get the resolution way down for some of these shaders. I also want to try to run them on pixel-LED screens.
I've been setting the sprite resolution to 20 x 20 px - then using normalized texcoords of the sprite in the shader code. But I just get the same performance as always, because I guess everything becomes nice and smooth and...normalized... so my method of thinking doesn't work?
Kind regards // Med venlig hilsen
New Media Artist // Creative Technologist
--
You received this message because you are subscribed to a topic in the Google Groups "pi3d" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pi3d/sQ8amvVqtO4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pi3d+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pi3d/7ab19e0b-d2b4-4c7a-9818-07bdebbce228o%40googlegroups.com.
Paddy
Jul 27, 2020, 2:46:37 AM7/27/20
to pi3d
Louise, I'd be interested to see how touchdesigner works. As far as I'm aware the fragment shader code will run for every pixel on the screen. The only way I know too do what you are trying to achieve its to render to a small area using a stencil, say, then redraw that texture full screen.
Will look at the touchdesigner code of it's available and post an example of drawing to reduced scale off screen textures.
Louise Lessél
Jul 27, 2020, 12:57:12 PM7/27/20
to Paddy, pi3d
Thank you very much Paddy. You are truly awesome.
And thank you for the code example - it works very well!
Yeah touchdesigner is a powerful tool - I use it for most of my installations - and when I VJ - but it would be much more beneficial to have things run on the pi (I can't be including a full spec computer with every art piece, its too expensive if I'm just running small LED screens haha)... and I've been wondering the same thing trying out different ways for days now, but I'm only familiar with the normalized scaling for shaders. I was hoping to maybe render smaller texture and then zoom in using the camera somehow.
Anyhow I also hope my tool will help people learn to create shaders, because touchdesigner is frankly very good at giving feedback on the shader code "uh oh.. something is wrong on line 45, we don't know that variable called c". I came into shaders from writing CG in Unity so that is quite a nice change!
Kind regards // Med venlig hilsen
New Media Artist // Creative Technologist
Den man. 27. jul. 2020 kl. 02.46 skrev Paddy <pat...@eldwick.org.uk>:
Louise, I'd be interested to see how touchdesigner works. As far as I'm aware the fragment shader code will run for every pixel on the screen. The only way I know too do what you are trying to achieve its to render to a small area using a stencil, say, then redraw that texture full screen.
Will look at the touchdesigner code of it's available and post an example of drawing to reduced scale off screen textures.
--
You received this message because you are subscribed to a topic in the Google Groups "pi3d" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pi3d/sQ8amvVqtO4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pi3d+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pi3d/7b3d34bd-01dc-4fad-b659-f9f24148c480o%40googlegroups.com.
Paddy
Jul 27, 2020, 5:56:45 PM7/27/20
to pi3d
Louise, I'm very relieved that the code worked. Yes shaders are a bit of a nightmare to debug so having a tool like that sounds attractive. Couldn't see any obvious links to the Touchdesigner source, as it's a commercial venture it could well be closed source (but didn't try too hard). The GPU on the Raspberry Pi is not so powerful and we had to tinker about quite a bit to get reasonable performance, some of the shadertoy examples are not really written in an efficient way, anyway this technique should speed it up. Please let me know if you have any issues.
pi3d had an OffScreenTexture class that really acts as a parent for derivative classes, PostProcess, Defocus, ShadowCaster, ClashTest but when I looked at it PostProcess already has the glScissor stuff in as well as a quad which it sets up and draws with the supplied shader so it's possible to just use that. The shadertoy01.fs code needs to have an extra factor to scale the pattern into the little rectangle. I randomly chose unif[4] (translates to unif[1][1] in the shader) to hold the scale value.
#include std_head_fs.inc
#define iTime unif[1][0]
#define iResolution unif[0]
#define scale unif[1][1]
void main(void) {
float t = iTime;
vec2 r = iResolution.xy;
float invScale = 1.0 / scale; // obviously scale must not be zero!
float offset = vec2(invScale - 1.0) * 0.5;
vec3 c;
float len, z=t;
for(int i=0; i<3; i++) {
}
gl_FragColor = vec4(c / len, t);
}The python code then needs to pass the scale value, define the PostProcessing instance, capture the render output then re-render it. For some reason I couldn't find a vanilla render from texture2D sampler to a quad so I made one and put it in the shaders/ folder. Python code:
import time
import demo
import pi3d
(W, H) = (None, None) # None should fill the screen (there are edge issues)
SCALE = 0.25 #should have 16th the shadertoy workload
display = pi3d.Display.create(w=W, h=H, frames_per_second=60.0)
print(display.opengl.gl_id)
if W is None or H is None:
(W, H) = (display.width, display.height)
sprite = pi3d.Triangle(corners=((-1.0, -1.0),(-1.0, 3.0),(3.0, -1.0)))
shader = pi3d.Shader('shaders/shadertoy01')
sprite.set_shader(shader)
## offscreen texture stuff ##
cam = pi3d.Camera(is_3d=False)
flatsh = pi3d.Shader('shaders/post_vanilla')
post = pi3d.PostProcess(camera=cam, shader=flatsh, scale=SCALE)
kbd = pi3d.Keyboard()
sprite.unif[0:2] = [W, H]
sprite.unif[4] = SCALE
tm0 = time.time()
while display.loop_running():
post.start_capture()
sprite.draw()
post.end_capture()
post.draw()
sprite.unif[3] = time.time() - tm0
k = kbd.read()
if k == 27:
breakshaders/post_vanilla.vs
#include std_head_vs.inc
varying vec2 texcoordout;
void main(void) {
texcoordout = texcoord * unib[2].xy + unib[3].xy;
texcoordout.y = 1.0 - texcoordout.y;
gl_Position = modelviewmatrix[1] * vec4(vertex, 1.0);
}shaders/post_vanilla.fs
#include std_head_fs.inc
varying vec2 texcoordout;
void main(void) {
gl_FragColor = texture2D(tex0, texcoordout);
gl_FragColor.a = 1.0;
}Paddy
Jul 27, 2020, 6:02:46 PM7/27/20
to pi3d
Wot! The google groups interface is beyond crap. It's just truncated by post.This is what is missing (I will do all the python again)Louise Lessél
Jul 28, 2020, 4:23:49 PM7/28/20
to Paddy, pi3d
Hi again and thank you again.
This time I'm getting an error in the shell output window.
Assuming I have copy pasted all of those files down correctly: https://github.com/louiselessel/Shaders-on-raspberry-pi4/tree/master/New%20Test2
Kind regards // Med venlig hilsen
New Media Artist // Creative Technologist
--
You received this message because you are subscribed to a topic in the Google Groups "pi3d" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pi3d/sQ8amvVqtO4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pi3d+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pi3d/d0867e18-1605-42b8-a2cf-4700c4c7ef63o%40googlegroups.com.
Louise Lessél
Jul 28, 2020, 4:30:38 PM7/28/20
to Paddy, pi3d
Oh I should mention that I tried both directly from the folder, as well as with the placing the Vanilla and touchdesigner shaders in the shaders folder.
Adjusting the lines for the shader like:
flatsh = pi3d.Shader('post_vanilla')
But I got that error message.
Kind regards // Med venlig hilsen
New Media Artist // Creative Technologist
Paddy
Jul 29, 2020, 4:35:58 AM7/29/20
to pi3d
Louise, sorry about that. I'm a bit surprised that it ran no problem on my laptop - obviously the mesa GLES drivers do a bit of hand-holding behind the scenes. Also your error message (thanks to Thonny, I might start using that to debug shader problems!!) was useful as it pointed to the fact that the seg fault was coming from the sprite.draw() line i.e. the shadertoy01 shader not the new offscreen capture system (if you comment all that out you will see it still doesn't work).
The culprit is the line in shadertoy01.fs
float offset = vec2(...
which should obviously be
vec2 offset = vec2(...
Paddy
Louise Lessél
Jul 29, 2020, 12:01:00 PM7/29/20
to Paddy, pi3d
Thank you so much!
It works wonderfully! I'll update my touchdesigner patch.
Kind regards // Med venlig hilsen
New Media Artist // Creative Technologist
--
You received this message because you are subscribed to a topic in the Google Groups "pi3d" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pi3d/sQ8amvVqtO4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pi3d+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pi3d/aff764cf-acab-4435-b631-ac8b1094b52fn%40googlegroups.com.
Louise Lessél
Jul 31, 2020, 12:45:55 PM7/31/20
to Paddy, pi3d
Still working on the automated touchdesigner patch to adapt to the new example (about 2/3 done), but i've been able to test out the code with a few shaders now and it runs very nicely!
I have a question about starting the program from the command line though, because usually when I do sudo python3... etc.. the file would open but it appears to me that opening it through there maybe causes the code to look for the shaders in the wrong place? (they are in the same folder as the .py script, no subfolders).
I get this error:
sudo python3 /home/pi/Desktop/Github/Shaders-on-raspberry-pi4/NewTest3shadertoy/run_shader.py
GL (NOTE: this is your printout from the python code, but then upon loading the shader I get this:)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/pi3d/Shader.py", line 192, in _load_shader
st = resource_string('pi3d', 'shaders/' + sfile)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1157, in resource_string
self, resource_name
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1401, in get_resource_string
return self._get(self._fn(self.module_path, resource_name))
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1587, in _get
with open(path, 'rb') as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.7/dist-packages/pi3d/shaders/shadertoy01.vs'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pi/Desktop/Github/Shaders-on-raspberry-pi4/NewTest3shadertoy/run_shader.py", line 20, in <module>
shader = pi3d.Shader('shadertoy01')
File "/usr/local/lib/python3.7/dist-packages/pi3d/Shader.py", line 149, in create
shader = Shader(shfile, vshader_source, fshader_source)
File "/usr/local/lib/python3.7/dist-packages/pi3d/Shader.py", line 110, in __init__
vshader_source, '.vs', GL_VERTEX_SHADER)
File "/usr/local/lib/python3.7/dist-packages/pi3d/Shader.py", line 97, in make_shader
src = src or self._load_shader(shfile + suffix, shader_type)
File "/usr/local/lib/python3.7/dist-packages/pi3d/Shader.py", line 195, in _load_shader
st = open(sfile, 'rb').read() #assume it's a non-standard shader in a file
FileNotFoundError: [Errno 2] No such file or directory: 'shadertoy01.vs'
Have you been able to run from commandline directly like that?
I can cd into the correct folder and then python3 run_shader.py just fine, but I'm trying to set it up in the auto boot using one of these methods https://www.dexterindustries.com/howto/run-a-program-on-your-raspberry-pi-at-startup/
Kind regards // Med venlig hilsen
New Media Artist // Creative Technologist
Louise Lessél
Aug 25, 2020, 3:57:50 PM8/25/20
to Paddy Ganut, pi3d
Hi Paddy
I hope you are well and thank you for all of the help so far. The resource is shaping up nicely!
I'm still working on the autoboot (or rather I postponed it for now).
I'm done with implementing the shadertoy converter in touchdesigner, so they all can run on the pi now! At the click of a button. Yay.
I am also almost done with implementing all of the uniforms.
However I have two quick questions, before I release it all to the world.
1. The mouse that I'm getting from pi3d.Mouse() appears to 1.) take the coordinates of the whole screen, not the display.
and 2.) when I take the mouse to the edge x or y positions, they are not always 0 and 1920. Sometimes they are like 344... and 1700...
and then if I keep dragging the mouse at the edge it becomes 0 or 1920... (the screen resolution). Is that a bug?
(full code: https://github.com/louiselessel/Shaders-on-raspberry-pi4/blob/master/Ex_All_Uniforms/run_shader_w_all_uniforms.py)
mouse = pi3d.Mouse() # pi3d.Mouse(restrict = True) # changes input coordinates mouse.start() MX, MY = mouse.position()
2. The iTimeDelta is a variable that tells me how long it took to draw the last frame, so like time since last frame, or rendertime.
Is there a way to get that number from somewhere?
3. And then I have a new question to make everything even more awesome:
I'm trying to implement an example that runs on a adafruit led pixel screen. (https://learn.adafruit.com/adafruit-rgb-matrix-plus-real-time-clock-hat-for-raspberry-pi/driving-matrices)
It uses PIL to set the image, see pasted code below. The reference here https://pillow.readthedocs.io/en/stable/reference/Image.html#examples
I'm wondering if there is a way to get the image directly from the buffer (from post), and set it to image?
I can see that you are already using PIL here https://pi3d.github.io/html/_modules/pi3d/Texture.html.
So I thought you might know?
Otherwise I'll ask the library master... hzeller (https://github.com/hzeller/rpi-rgb-led-matrix/tree/master/bindings/python)
(full code here https://github.com/louiselessel/Shaders-on-raspberry-pi4/blob/master/Work-in-progress/image-draw-shader.py)
The full code currently draws the shader to the display on my computer screen, and it draws a rectangle with a cross on my adafruit led screen. But I want it to just draw the shader, so I need it as PIL.
#-------------------------------------------------
# PUT SHADER ON MATRIX
while display.loop_running():
post.start_capture()
sprite.draw()
post.end_capture()
post.draw()
sprite.unif[3] = time.time() - tm0
# draw the shader buffer into a PIL image here <------------ #
# set image to matrix
matrix.SetImage(image,0,0)#-------------------------------------------------
Please also do have a look at the attribution I made in the bottom:
I made one postprocessing shader so far, a pixelize shader, which I did have to grapple a little with the weird edge issues (from the downscaling) for,
because I was sampling right at the edge XD
But I made it work!!! Very proud!
Ugh. Long email, sorry!
Kind regards // Med venlig hilsen
New Media Artist // Creative Technologist
Den fre. 31. jul. 2020 kl. 13.24 skrev Paddy Ganut <paddy...@gmail.com>:
Louise, I always run directly from the command line. It generally seems to turn out easier! I do all the development on Ubuntu laptop with MS code editor.The argument passed to Shader() is treated as an absolute path if it starts / otherwise it is treated as relative to the program location otherwise it is looked for in pi3d install directory shaders/ it uses normal os file functions but adds .vs and .fsSo tldr; shaders in same directory don't need shaders in name.Paddy
To view this discussion on the web visit https://groups.google.com/d/msgid/pi3d/CAFeYzYNe8uVkVGov1D5AGhXyPc28wVO0VS8cg0LTc%3DM03DtUhw%40mail.gmail.com.
Paddy
Aug 25, 2020, 5:33:48 PM8/25/20
to pi3d
Louise, sounds good progress. I will look at your questions properly tomorrow but very briefly. 1) The pi3d mouse depends a bit on where pi3d is being run - if it's on the RPi4 with the fake KMS driver then it's using the x11 mouse. This does various tweaks to make it behave in as similar way as possible to the curses mouse used with the non x version of raspbian. I will look at the code and figure out what combination you need to get the mouse location in the window. 2) The Display instance must know what time each frame was rendered in order to throttle fps, again I will look where that it is and let you know. 3) there is a glReadPixels() function that is available on OpenGLES2 and which pi3d uses in the screenshot() function. I'm not sure if it can be used to read from the post buffer. In your code it would be something like
image = Image.frombuffer('RGB', (w, h), pi3d.screenshot(), 'raw', 'RGB', 0, 1)
But this obviously has a few overheads creating and copying data from place to place. I will see if I can find a way to do it a bit more efficiently.
Paddy
Louise Lessél
Aug 25, 2020, 5:55:16 PM8/25/20
to Paddy, pi3d
Great thank you!
Yeah I thought there might be something like a read pixels() :) but yeah as you say maybe there’s an overhead there. Though it might not be noticeable at a resolution of 32 x 32. Maybe only if I begin chaining more screens.
I’ll try it tomorrow if you haven’t already sent something by the time I get up over here in New York :)
Thank you again!
To view this discussion on the web visit https://groups.google.com/d/msgid/pi3d/6bb15c8e-fe26-46b0-9de0-158c1ff6b8cdn%40googlegroups.com.
Paddy
Aug 26, 2020, 1:45:47 PM8/26/20
to pi3d
I might add this extra functionality to pi3d.screenshot() but at the moment I'm having to do this, which is essentially just copied from screenshot.
import numpy as np
from PIL import Image
import ctypes
from pi3d.constants import (opengles, GLint, GLsizei, GLubyte, GL_DEPTH_COMPONENT,
GL_RGBA, GL_UNSIGNED_BYTE, GL_UNSIGNED_SHORT, PIL_OK)
...
sprite.unif[0:2] = [W, H]
sprite.unif[4] = SCALE
tm0 = time.time()
f = 0
(ws, hs) = (int(W * SCALE), int(H * SCALE))
(xos, yos) = (int((W - ws) * 0.5), int((H - hs) * 0.5))
np_img = np.zeros((hs, ws, 4), dtype=np.uint8)
while display.loop_running():
f += 1
post.start_capture()
sprite.draw()
if f == 50: # just do it once for testing
opengles.glReadPixels(GLint(xos), GLint(yos), GLsizei(ws), GLsizei(hs), GL_RGBA,
GL_UNSIGNED_BYTE, np_img.ctypes.data_as(ctypes.POINTER(GLubyte)))
np_img = np_img[::-1,:,:3].copy()
a = Image.frombuffer("RGB", (ws, hs), np_img,'raw', 'RGB', 0, 1)
post.end_capture()
...
Paddy
Aug 26, 2020, 1:54:50 PM8/26/20
to pi3d
Louise, also there is a variable Display.time which is set at the start of the frame. To determine the delta time of the previous frame I suppose you would need to
dt = display.time - last_time
...
last_time = display.timeI will look a bit more at the mouse and screenshot code and get back to you.
Paddy
Louise Lessél
Aug 28, 2020, 12:43:01 PM8/28/20
to Paddy, pi3d
Thank you Paddy.
I'll test it today. Time got away from me yesterday.
Kind regards // Med venlig hilsen
New Media Artist // Creative Technologist
@louiselessel
Artist inquiries: lou...@louiselessel.com
NYU inquiries: louise...@nyu.edu
Private: louise...@gmail.com
--
You received this message because you are subscribed to a topic in the Google Groups "pi3d" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pi3d/sQ8amvVqtO4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pi3d+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pi3d/35b7e2ea-6878-4e46-af5a-cd2075022756o%40googlegroups.com.
Paddy
Aug 28, 2020, 12:53:21 PM8/28/20
to pi3d
Louise, I've just realised that taking multiple frames will cause issues as the numpy array is overwritten with a different sized one! I will post the correction in a few minutes. I've also modified screenshot which I will also post soon.
Paddy
Paddy
Aug 28, 2020, 2:35:42 PM8/28/20
to pi3d
Louise, I've pushed the changes to pi3d develop branch. There's now a function in pi3d/util/Screenshot.py called masked_screenshot() that can be called to get a sub-rectangle of the display buffer. My screenshot01.py file has become:
import time
import demo
import pi3d
from PIL import Image
(W, H) = (None, None) # None should fill the screen (there are edge issues)
SCALE = 0.25 #should have 16th the shadertoy workload
display = pi3d.Display.create(w=W, h=H, frames_per_second=60.0)
print(display.opengl.gl_id)
if W is None or H is None:
(W, H) = (display.width, display.height)
sprite = pi3d.Triangle(corners=((-1.0, -1.0),(-1.0, 3.0),(3.0, -1.0)))
shader = pi3d.Shader('shaders/shadertoy01')
sprite.set_shader(shader)
## offscreen texture stuff ##
cam = pi3d.Camera(is_3d=False)
flatsh = pi3d.Shader('shaders/post_vanilla')
post = pi3d.PostProcess(camera=cam, shader=flatsh, scale=SCALE)
kbd = pi3d.Keyboard()
sprite.unif[0:2] = [W, H]
sprite.unif[4] = SCALE
tm0 = time.time()
f = 0
(ws, hs) = (int(W * SCALE), int(H * SCALE))
(xos, yos) = (int((W - ws) * 0.5), int((H - hs) * 0.5))
while display.loop_running():
f += 1
post.start_capture()
sprite.draw()
if f == 50 or f == 51:
a = Image.fromarray(pi3d.masked_screenshot(xos, yos, ws, hs))
post.end_capture()
post.draw()
sprite.unif[3] = time.time() - tm0
k = kbd.read()
if k == 27:
kbd.close()
display.stop()
breakI will try to push a new release of pi3d master branch tonight but the change to Screenshot is
def masked_screenshot(x, y, w, h):
"""returns numpy array from part of screen so it can be used by applications
drawing low resolution offscreen textures using scaling.
"""
img = np.zeros((h, w, 4), dtype=np.uint8)
opengles.glReadPixels(GLint(x), GLint(y), GLsizei(w), GLsizei(h), GL_RGBA,
GL_UNSIGNED_BYTE, img.ctypes.data_as(ctypes.POINTER(GLubyte)))
return img[::-1,:,:3].copy()Obviously there is a bit of inefficiency creating the (h, w, 4) array then copying it to a (h, w, 3) array if lots of frames are to be taken then the same array could be overwritten by glReadPixels() but the copy has to be take otherwise PIL scrambles the images if it's just a view into the original array. Ideally I would read just RGB but this doesn't seem to work for some reason...
Paddy
Louise Lessél
Aug 28, 2020, 9:25:25 PM8/28/20
to Paddy, pi3d
This is great. I see the merge request!
I was able to implement the iTimeDelta as well.
And I was able to get the rgb matrix implemented with the screenshot() for now, I have prepared it to run with the masked_screenshot as well, when the merge happens.
Another question, because I am getting a "No module named pi3d" when I run from the terminal (it is fine when I run by pressing Run in Thonny however)...
which I think is probably what's stopping my autoboot from working also.
Is there a correct way to run the code from terminal, so it imports?
The import order is the same as yours above.
Kind regards // Med venlig hilsen
New Media Artist // Creative Technologist
@louiselessel
Artist inquiries: lou...@louiselessel.com
NYU inquiries: louise...@nyu.edu
Private: louise...@gmail.com
--
You received this message because you are subscribed to a topic in the Google Groups "pi3d" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pi3d/sQ8amvVqtO4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pi3d+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pi3d/b888f3b2-68be-4555-a641-e8b9ae68b5d2o%40googlegroups.com.
Paddy
Aug 29, 2020, 4:03:21 AM8/29/20
to pi3d
Louise, I pushed the latest version to master and pypi.python.org so `pip3 install pi3d --upgrade` should allow pi3d.masked_screenshot() to work now.
I'm not entirely sure about your error but it might be related to the fact that you use python rather than python3. I think Thonny and many other apps now default to python3 (as they should). I've tried to persuade the RPi foundation that they should diverge from Debian here and make python === python3 or at least make the commandline shell generate a warning if you type python - it seems mad to allow it to revert to 2 (I no longer do the full testing for pi3d on python2). Let me know if there is still a problem and I will think further.
If you have a local version of the pi3d repository on your computer (i.e. from git clone) you can make demo.py point to that and import demo before you import pi3d which is what I do here with development.
I'm doing things over the next few days so not sure when I will get a chance to look at the mouse location but will let you know.
Paddy
Paddy
Aug 30, 2020, 6:45:16 AM8/30/20
to pi3d
Louise, I just had a quick look at the pi3d.Mouse class and I see that there is an optional argument you can use on x11 windows
mouse = pi3d.Mouse(use_x=True)
which then make mouse.position() return the mouse location relative to the centre of the x11 window. If you move the mouse off the edge of the window the value will continue to change until you get to the edge of the screen so if you wanted to clamp it you would have to min(W * 0.5, max(-W * 0.5, mx)) kind of thing
Louise Lessél
Sep 1, 2020, 8:50:15 AM9/1/20
to Paddy, pi3d
Great,
I will try that out.
1)
In the meantime, I have been trying out the masked screenshot.
The edge issue becomes apparent when I run the shader at a small resolution such as 32x32 (the adafruti pixelscreen resolution).
I made the background behind the shader red so it's easier to spot when the edge issue occurs. Here I am running it at 0.8 SCALE.
You can see the tiny 32x32 shader to the left on the computer screen.
The images are of the output to the adafruit screen.
(everything works nicely at 1.0 SCALE)
2)
I managed to download the pi3d and point to it in demo.py. Now i'm getting a "X11 needs to be running?"
Kind regards // Med venlig hilsen
New Media Artist // Creative Technologist
@louiselessel
Artist inquiries: lou...@louiselessel.com
NYU inquiries: louise...@nyu.edu
Private: louise...@gmail.com
--
You received this message because you are subscribed to a topic in the Google Groups "pi3d" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pi3d/sQ8amvVqtO4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pi3d+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pi3d/53b2ba85-e262-437c-b89f-89426d2b5aa7o%40googlegroups.com.
Paddy
Sep 1, 2020, 11:43:30 AM9/1/20
to pi3d
Louise, I will look at the 32x32 edge issue later - I'm not entirely sure what I'm looking at. On the RPi 4 pi3d needs to have the X server running to get a render surface. If you don't want the whole desktop running then you can start just the required bit with xinit. Though it does look like you have the desktop running in you picture, maybe send me details of exactly what you're doing.
Paddy
PS it's interesting to see the march of heif/heic image format, fine for looking at images in browsers or on apple kit, but a bit of catching up needed elsewhere (I can't get them to import to PIL on the RPi yet!)
Louise Lessél
Sep 21, 2020, 4:59:46 PM9/21/20
to Paddy, pi3d
Hi Paddy.
I hope you are well!
I've been a little busy on other projects. But I finished the tool!
Try it out! I'm gonna make a post to the touchdesigner community later this week about it.
Also figured out the run on boot up. And added an example for running the shaders on the adafruit matrix:
And the touchdesigner conversion tool is here:
-----
There are still edge issues with the scaling method. Like you initially commented in the code you gave me. Do you know what is causing it? Because when sampling for the postprocessing shaders (like the pixel shader) it does make things a little difficult - if sampling at the edge, I end up making the edge issue BIGGER, since I'm samling the edge, not the shader area... haha.
I had to move the sampling area in from the edge to avoid this.
For the matrix, I think the problem is that the smaller the display the more apparent the issue. So for the matrix at 32 x 32, when using the SCALE = 0.4, the edge issue is very visible. At SCALE = 1.0 there is of course no issue. When the size of the display is say 400 x 400 the edge issue is so small (almost invisible at the edge).
What I was saying is that I have been making the background color be red, so it's easier to see.
-----
Also. I'm wondering how you are doing this?
It would be cool to get a video file in, and a live cam.
Do you have an example already from this that I could take apart?
All of the pi3d examples have so many different things going on!
They are a little hard to take apart haha.
Kind regards // Med venlig hilsen
New Media Artist // Creative Technologist
@louiselessel
Artist inquiries: lou...@louiselessel.com
NYU inquiries: louise...@nyu.edu
Private: louise...@gmail.com
--
You received this message because you are subscribed to a topic in the Google Groups "pi3d" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pi3d/sQ8amvVqtO4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pi3d+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pi3d/ea08c46f-eeee-4039-b6ff-1a655f888725o%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages