[fog-dev] CG Shader Parser Design
Mike Tajmajer
The ARB Compiler Design is very simple.
1) The context structure is passed into the fragment and is used for scratch
storage, and persistent data.
The fragment may be called once, or multiple times during a rendering.
It is in flux, as I am not happy about how run-time PARAMs are handled -- no
please keep that in mind.
The result and fragment are not yet complete.
namespace ARBCompiler
{
// ARB Context - pointer is passed into function
// be sure that the data is properly aligned
struct Context
{
// Types
typedef std::vector<std::string> StringVec;
// 'result' Param
__declspec(align( 16 )) struct result
{
float color[4];
} result_;
// 'fragment' Param
__declspec(align( 16 )) struct fragment
{
float color[4];
} fragment_;
// Temp Data
__declspec(align( 16 )) float xmmTemp_[4];
__declspec(align( 16 )) float xmmTemp2_[4];
long i32Temp_;
// Persistent Params
unsigned nParams_;
static const unsigned nMaxParams_ = 32;
StringVec vecParamNames_;
__declspec(align( 16 )) float arParams_[nMaxParams_*4];
char arParamInitFlag_[nMaxParams_];
Context(StringVec paramNames)
{
xmmTemp_[0] = 0.0;
xmmTemp_[1] = 0.0;
xmmTemp_[2] = 0.0;
xmmTemp_[3] = 0.0;
xmmTemp2_[0] = 0.0;
xmmTemp2_[1] = 0.0;
xmmTemp2_[2] = 0.0;
xmmTemp2_[3] = 0.0;
i32Temp_ = 0;
result_.color[0] = 0.0;
result_.color[1] = 0.0;
result_.color[2] = 0.0;
result_.color[3] = 1.0;
fragment_.color[0] = 0.0;
fragment_.color[1] = 0.0;
fragment_.color[2] = 0.0;
fragment_.color[3] = 1.0;
// index of symbol name in the vector defines the XMMRef and
float pointer
nParams_ = paramNames.size();
assert(nParams_ <= nMaxParams_);
vecParamNames_ = paramNames;
for(unsigned i=0;i<nMaxParams_;++i)
{
arParamInitFlag_[i] = 0;
arParams_[i*4+0] =
arParams_[i*4+1] =
arParams_[i*4+2] =
arParams_[i*4+3] = 0.0;
}
}
~Context()
{
}
bool isParam(const std::string & str, unsigned & nOffset)
{
unsigned n(0);
StringVec::iterator itr;
for(itr=vecParamNames_.begin();itr!=vecParamNames_.end();++itr,++n)
{
if(0 == str.compare((*itr)))
{
nOffset = (char*)&(arParams_[n*4]) - (char*)this;
return true;
}
}
return false;
}
};
// Function Type
typedef void (*fragFP)(Context *);
// Parse the ARB Code Block, return true if success, with number and
names of PARAMs
extern bool getParams(const std::string & strARB,
std::vector<std::string> & vecParamNames);
// Compile the ABR to a function - to use, call the fp with a Context
pointer
extern bool compileARB(const std::string & strARB, const
std::vector<std::string> & vecParamNames, fragFP & fpFrag);
// Release the Fragment
extern void freeFragment(fragFP fpFrag);
}
2) The parser is based on the YARD parser library:
http://code.google.com/p/yardparser. This is public domain code.
I have modified it to not use RTTI, and also extent the maximum length of
the symbols.
I have attached a zip file of the modified code.
3) The grammar is not complete yet. I need to extend the PARAM to handle
arrays of PARAMS.
You will see that the rules are identified by an enum that is stored as a
class const.
namespace ARBCompiler
{
using namespace yard;
using namespace text_grammar;
// Parser Types
typedef yard::SimpleTextParser Parser;
typedef Parser::Node Node;
// Rule ID's
const enum Rule_Types
{
ARB_WhiteSpace = 0,
ARB_StringCharLiteral,
ARB_CharLiteral,
ARB_StringLiteral,
ARB_BinaryDigit,
ARB_BinNumber,
ARB_HexNumber,
ARB_DecNumber,
ARB_IntNumber,
ARB_Tok,
ARB_ID,
ARB_CharTok,
ARB_Bracketed,
ARB_ARBComment,
ARB_ARBHeader,
ARB_ARBEnd,
ARB_Vector,
ARB_Keyword,
ARB_PARAM,
ARB_TEMP,
ARB_MUL,
ARB_ADD,
ARB_ABS,
ARB_MOV,
ARB_TEX,
ARB_OP_1D,
ARB_OP_2D,
ARB_OP_3D,
ARB_TEXOP,
ARB_DP3,
ARB_POW,
ARB_MIN,
ARB_MAX,
ARB_DOT_OP,
ARB_NEG_OP,
ARB_ArrayIndex,
ARB_AssignOp,
ARB_PostfixSuffix,
ARB_InputExp,
ARB_OutputExp,
ARB_EOS,
ARB_Opcode2,
ARB_Opcode3,
ARB_OpcodeTEX,
ARB_ParamStatement,
ARB_TempStatement,
ARB_ExecuteStatement,
ARB_ARBStatement,
ARB_ARBStatementList,
ARB_File,
ARB_ResultDotColor,
ARB_Result,
ARB_FragmentDotColor,
ARB_FragmentDotColorDotPrimary,
ARB_Fragment,
ARB_BuiltInParams,
};
const std::string arRuleName[] =
{
"ARB_WhiteSpace",
"ARB_StringCharLiteral",
"ARB_CharLiteral",
"ARB_StringLiteral",
"ARB_BinaryDigit",
"ARB_BinNumber",
"ARB_HexNumber",
"ARB_DecNumber",
"ARB_IntNumber",
"ARB_Tok",
"ARB_ID",
"ARB_CharTok",
"ARB_Bracketed",
"ARB_ARBComment",
"ARB_ARBHeader",
"ARB_ARBEnd",
"ARB_Vector",
"ARB_Keyword",
"ARB_PARAM",
"ARB_TEMP",
"ARB_MUL",
"ARB_ADD",
"ARB_ABS",
"ARB_MOV",
"ARB_TEX",
"ARB_OP_1D",
"ARB_OP_2D",
"ARB_OP_3D",
"ARB_TEXOP",
"ARB_DP3",
"ARB_POW",
"ARB_MIN",
"ARB_MAX",
"ARB_DOT_OP",
"ARB_NEG_OP",
"ARB_ArrayIndex",
"ARB_AssignOp",
"ARB_PostfixSuffix",
"ARB_InputExp",
"ARB_OutputExp",
"ARB_EOS",
"ARB_Opcode2",
"ARB_Opcode3",
"ARB_OpcodeTEX",
"ARB_ParamStatement",
"ARB_TempStatement",
"ARB_ExecuteStatement",
"ARB_ARBStatement",
"ARB_ARBStatementList",
"ARB_File",
"ARB_ResultDotColor",
"ARB_Result",
"ARB_FragmentDotColor",
"ARB_FragmentDotColorDotPrimary",
"ARB_Fragment",
"ARB_BuiltInParams",
};
namespace arb_parser
{
//
// Used to store the Rule Type in the AST
//
template<class Rule_T>
struct Store
{
// Note: due to what seems like a VC compiler
bug, this code must be inlined
template<typename ParserState_T>
static bool Match(ParserState_T& p)
{
bool f(false);
#ifdef PARSER_DEBUG
std::string str(p.GetPos(), p.GetPos()+80);
int c = str.find("\n");
if(std::string::npos != c)
str = std::string(str.begin(),
str.begin()+c);
str = trim(str);
printf("try node %s [%s]\n",
typeid(Rule_T).name(), str.c_str());
#endif
p.template CreateNode<Rule_T>();
try
{
f = Rule_T::template Match(p);
}
catch(...)
{
p.AbandonNode();
throw;
}
if(f)
{
p.CompleteNode(Rule_T::type());
#ifdef PARSER_DEBUG
printf("Match %s\n",
typeid(Rule_T).name());
#endif
return true;
}
p.AbandonNode();
return false;
}
};
//
// Forward def's
//
struct ARBComment;
//
// Parser Rules
//
struct NewLine
: Or<CharSeq<'\r', '\n'>, CharSeq<'\n'> >
{
};
struct WhiteSpace
: Star<Or<Char<' '>, Char<'\t'>, NewLine,
ARBComment, Char<'\r'>, Char<'\v'>, Char<'\f'> > >
{
};
/*
struct StringCharLiteral
: Or<Seq<Char<'\\'>, NotChar<'\n'> >,
AnyCharExcept<CharSet<'\n', '\"', '\'' > > >
{
};
struct CharLiteral
: Seq<Char<'\''>, StringCharLiteral, Char<'\''> >
{
};
struct StringLiteral
: Seq<Char<'\"'>, Star<StringCharLiteral>,
Char<'\"'> >
{
};
struct BinaryDigit
: Or<Char<'0'>, Char<'1'> >
{
};
struct BinNumber
: Seq<CharSeq<'0', 'b'>, Plus<BinaryDigit>,
NotAlphaNum, WhiteSpace>
{
};
struct HexNumber
: Seq<CharSeq<'0', 'x'>, Plus<HexDigit>,
NotAlphaNum, WhiteSpace>
{
};
*/
struct DecNumber
: Seq<Opt<Char<'-'> >, Plus<Digit>,
Opt<Seq<Char<'.'>, Star<Digit> > >, NotAlphaNum, WhiteSpace>
{
RULE_TYPE(ARB_DecNumber);
};
struct IntNumber
: Seq<Opt<Char<'-'> >, Plus<Digit>,
Opt<Star<Digit>>, NotAlphaNum, WhiteSpace>
{
RULE_TYPE(ARB_IntNumber);
};
template<typename R>
struct Tok
: Seq<R, WhiteSpace>
{
};
struct ID
: Tok<Ident>
{
RULE_TYPE(ARB_ID);
};
template<char C>
struct CharTok
: Seq<Char<C>, WhiteSpace>
{
};
template<typename R>
struct Bracketed
: Seq<CharTok<'['>, Store<R>, CharTok<']'> >
{
RULE_TYPE(ARB_Bracketed);
};
struct ARBComment
: Seq<Char<'#'>, UntilPast<NewLine>, WhiteSpace>
{
};
struct ARBHeader
:
Seq<CharSeq<'!','!','A','R','B','f','p','1','.','0'>, UntilPast<NewLine>,
WhiteSpace>
{
};
struct ARBEnd
: Seq<CharSeq<'E','N','D'>, UntilPast<NewLine>,
WhiteSpace>
{
RULE_TYPE(ARB_ARBEnd);
};
struct Vector
: Seq<CharTok<'{'>, Store<DecNumber>, CharTok<','>,
Store<DecNumber>, CharTok<','>, Store<DecNumber>, CharTok<','>,
Store<DecNumber>, CharTok<'}'>, WhiteSpace>
{
RULE_TYPE(ARB_Vector);
};
template<typename T>
struct Keyword :
Tok<Seq<T, NotAlphaNum > >
{
};
struct PARAM
: Keyword<CharSeq<'P','A','R','A','M'>>
{
RULE_TYPE(ARB_PARAM);
};
struct TEMP
: Keyword<CharSeq<'T','E','M','P'>>
{
RULE_TYPE(ARB_TEMP);
};
struct MUL
: Keyword<CharSeq<'M','U','L'>>
{
RULE_TYPE(ARB_MUL);
};
struct ADD
: Keyword<CharSeq<'A','D','D'>>
{
RULE_TYPE(ARB_ADD);
};
struct ABS
: Keyword<CharSeq<'A','B','S'>>
{
RULE_TYPE(ARB_ABS);
};
struct MOV
: Keyword<CharSeq<'M','O','V'>>
{
RULE_TYPE(ARB_MOV);
};
struct TEX
: Keyword<CharSeq<'T','E','X'>>
{
RULE_TYPE(ARB_TEX);
};
struct OP_1D
: Keyword<CharSeq<'1','D'>>
{
RULE_TYPE(ARB_OP_1D);
};
struct OP_2D
: Keyword<CharSeq<'2','D'>>
{
RULE_TYPE(ARB_OP_2D);
};
struct OP_3D
: Keyword<CharSeq<'3','D'>>
{
RULE_TYPE(ARB_OP_3D);
};
struct TEXOP
: Or<Store<OP_1D>,Store<OP_2D>,Store<OP_3D>>
{
RULE_TYPE(ARB_TEXOP);
};
struct DP3
: Keyword<CharSeq<'D','P','3'>>
{
RULE_TYPE(ARB_DP3);
};
struct POW
: Keyword<CharSeq<'P','O','W'>>
{
RULE_TYPE(ARB_POW);
};
struct MIN
: Keyword<CharSeq<'M','I','N'>>
{
RULE_TYPE(ARB_MIN);
};
struct MAX
: Keyword<CharSeq<'M','A','X'>>
{
RULE_TYPE(ARB_MAX);
};
struct DOT_OP
: Tok<Char<'.'> >
{
RULE_TYPE(ARB_DOT_OP);
};
struct NEG_OP
: Tok<Char<'-'> >
{
RULE_TYPE(ARB_NEG_OP);
};
struct ArrayIndex :
Bracketed<IntNumber>
{
RULE_TYPE(ARB_ArrayIndex);
};
struct ResultDotColor
:
Keyword<CharSeq<'r','e','s','u','l','t','.','c','o','l','o','r'>>
{
RULE_TYPE(ARB_ResultDotColor);
};
struct Result
: Store<ResultDotColor>
{
RULE_TYPE(ARB_Result);
};
// Same as FragmentDotColor
struct FragmentDotColorDotPrimary
:
Keyword<CharSeq<'f','r','a','g','m','e','n','t','.','c','o','l','o','r','.',
'p','r','i','m','a','r','y'>>
{
RULE_TYPE(ARB_FragmentDotColorDotPrimary);
};
struct FragmentDotColor
:
Keyword<CharSeq<'f','r','a','g','m','e','n','t','.','c','o','l','o','r'>>
{
RULE_TYPE(ARB_FragmentDotColor);
};
struct Fragment
:
Or<Store<FragmentDotColorDotPrimary>,Store<FragmentDotColor>>
{
RULE_TYPE(ARB_Fragment);
};
struct BuiltInParams
: Or<Fragment,Result>
{
RULE_TYPE(ARB_BuiltInParams);
};
struct AssignOp
: Seq<Tok<Seq<Char<'='>, NotAt<Char<'='> > > >,
WhiteSpace>
{
RULE_TYPE(ARB_AssignOp);
};
struct PostfixSuffix
: Or<Store<ArrayIndex>, Seq<Store<DOT_OP>,
Store<ID>>>
{
RULE_TYPE(ARB_PostfixSuffix);
};
struct InputExp
: Or<
Store<DecNumber>,
Seq<BuiltInParams,
Opt<Store<PostfixSuffix>>>,
Seq<Opt<Store<NEG_OP>>, Store<ID>,
Star<Store<PostfixSuffix>>>,
Store<Vector>
>
{
RULE_TYPE(ARB_InputExp);
};
struct OutputExp
: Or<
Seq<BuiltInParams, Opt<Store<PostfixSuffix>>>,
Seq<Store<ID>, Star<Store<PostfixSuffix>>>
>
{
RULE_TYPE(ARB_OutputExp);
};
struct EOS
: CharTok<';'>
{
};
struct Opcode2
: Or<Store<ABS>,Store<MOV>>
{
RULE_TYPE(ARB_Opcode2);
};
struct Opcode3
:
Or<Store<MUL>,Store<ADD>,Store<DP3>,Store<POW>,Store<MIN>,Store<MAX>>
{
RULE_TYPE(ARB_Opcode3);
};
struct OpcodeTEX
: Store<TEX>
{
RULE_TYPE(ARB_OpcodeTEX);
};
struct ParamStatement
: Seq<Store<PARAM>, Store<OutputExp>,
Store<AssignOp>, Or<Store<Vector>,Store<DecNumber>,Store<InputExp>>, EOS>
{
RULE_TYPE(ARB_ParamStatement);
};
struct TempStatement
: Seq<Store<TEMP>, Store<ID>,
Opt<Seq<Store<AssignOp>,
Or<Store<Vector>,Store<DecNumber>,Store<InputExp>>>>, EOS>
{
RULE_TYPE(ARB_TempStatement);
};
struct ExecuteStatement
: Or<
Seq<Store<Opcode2>, Store<OutputExp>, CharTok<','>,
Store<InputExp>, EOS>,
Seq<Store<Opcode3>, Store<OutputExp>, CharTok<','>,
Store<InputExp>, CharTok<','>, Store<InputExp>, EOS>,
Seq<Store<OpcodeTEX>, Store<OutputExp>,
CharTok<','>, Store<InputExp>, CharTok<','>, Store<InputExp>, CharTok<','>,
TEXOP, EOS>
>
{
RULE_TYPE(ARB_ExecuteStatement);
};
struct ARBStatement
: Or<Store<ParamStatement>, Store<TempStatement>,
Store<ExecuteStatement>>
{
RULE_TYPE(ARB_ARBStatement);
};
struct ARBStatementList
: Plus<ARBStatement>
{
RULE_TYPE(ARB_ARBStatementList);
};
struct File
: Seq<ARBHeader, ARBStatementList, Store<ARBEnd>,
EndOfInput>
{
RULE_TYPE(ARB_File);
};
}
}
4) The code generator operates on the AST tree. I parse the code twice, the
first time counts the PARAMs declared in the code, the second time it
generates the function.
I start by handling the top level statements:
void Compiler::CompileAST(Node *p, Compiler & comp)
{
if(!comp.isOK())
return;
Symbol *pVar(NULL);
switch(p->GetUserValue())
{
case ARB_ParamStatement:
printRule(p);
TraverseVar(p, 1, comp, pVar, true);
if(!comp.isOK())
return;
pVar->movLitVecToParamIfNotInit(comp.c_, comp.ctx_,
pVar->vec_, comp.getParamOffset(pVar->name_));
break;
case ARB_TempStatement:
// This will add a TEMP to the symbol table
printRule(p);
TraverseVar(p, 1, comp, pVar, false);
if(!comp.isOK())
return;
pVar->movLitVecToVar(comp.c_, comp.ctx_,
pVar->vec_);
break;
case ARB_ExecuteStatement:
printRule(p);
TraverseStatement(p, 1, comp);
break;
case ARB_ARBEnd:
printRule(p);
break;
default:
comp.ErrorState("Unknown Top Level Construct.");
break;
}
}
Each rule is handled in a compiler method (example of ADD):
void Compiler::ProcessADD(Node *p, int depth, Compiler & comp)
{
VarInfo dst;
Symbol *pTemp0(0);
Symbol *pTemp1(0);
Process3Arg(p, depth, comp, dst, pTemp0, pTemp1);
pTemp0->addVar(comp.c_, comp.ctx_, pTemp1);
if(dst.swizCount_> 0)
{
unsigned arSwiz[4] = {0, 1, 2, 3};
dst.pVar_->movToVarIdx(comp.c_, comp.ctx_, pTemp0,
dst.swizCount_, arSwiz, dst.arSwiz_, false);
}
else
{
dst.pVar_->movVarToVar(comp.c_, pTemp0);
}
}
The symbol object actually generates the code -- the swizzle is handled via
pshufd or by using a temp in the context.
void Symbol::addVar(AsmJit::Compiler & c, AsmJit::PtrRef & ctxRef, Symbol
*pSym)
{
c.comment("Add var %s to var %s", name_.c_str(),
pSym->name_.c_str());
c.addps(var_.r(), pSym->var_.c());
}
Petr Kobalíček
thanks for detailed post, some comments in paragraphs below:
I think that align(16) is not needed, we can alloc whole context 16-byte aligned, of course proper variable position is needed in that case (moving all aligned members to the top).
Also replacing std::vector by Fog::List and std::string by Fog::String will be needed, but I understand here that it's Fog free implementation at this time, good!
2) The parser is based on the YARD parser library:
http://code.google.com/p/yardparser. This is public domain code.
I have modified it to not use RTTI, and also extent the maximum length of
the symbols.
I have attached a zip file of the modified code.
Understand, but I'd like to remove yard dependency in the future, depending on patched code is thing I'd like to avoid. For now it's ok!
3) The grammar is not complete yet. I need to extend the PARAM to handle
arrays of PARAMS.
You will see that the rules are identified by an enum that is stored as a
class const.
How parameters are handled? I read some GLSL specs and there are more kind of parameters. I think that we should also add support for this so shader can be reused many times with different parameters. But on the other side constant parameters should be folded.
I not understand here, parsing twice? If you have AST then it's no problem to count variables, isn't it?
This is good.
The symbol object actually generates the code -- the swizzle is handled via
pshufd or by using a temp in the context.
Smart, I like pshufd.
void Symbol::addVar(AsmJit::Compiler & c, AsmJit::PtrRef & ctxRef, Symbol
*pSym)
{
c.comment("Add var %s to var %s", name_.c_str(),
pSym->name_.c_str());
c.addps(var_.r(), pSym->var_.c());
}
I try to play with code you post, if I found something strange I will inform you early;)
--
Best regards
- Petr Kobalicek <http://kobalicek.com>
Mike Tajmajer
About your comments:
The parsing twice is something I need to remove.
The PARAM data block needs more thought. I am learning more about how the
ARB code is consumed, and need to update how the PARAM data is handled.
Also, this will be how the fragments interact with the Fog Graphics
Pipeline.
About the next steps:
1) I need to add support for PARAM arrays and vector arrays (arrays of
vectors).
2) I need to add more OPCodes.
3) I need to continue to refactor and make the design cleaner. This will be
ongoing.
Now there are 3 classes, Compiler, Symbol, Context.
The code generation is part of the Symbol - I don't like that. I will
create a Code Generator class.
The Symbol class needs to be broken into an interface and child
implementations so that different types of symbols may be supported (TEMP,
PARAM, etc.)
4) Now the project just runs the test code -- and is just has a Windows
project file.
I need to make it an actual library, and have a cmake file for it.
Petr Kobalíček
I tried it yesterday night, but I didn't compile it (I haven't cg installed).
But I think that there is one big question: Do you want that shader code will depend to Fog or not? I mean that there is possibility to make it independent, but to be honest I'd like to make it Fog-only (because we can make better integration with the framework).
I'm going to dig it today. hope that there will be success;)
Hi Petr,
About your comments:
The parsing twice is something I need to remove.
The PARAM data block needs more thought. I am learning more about how the
ARB code is consumed, and need to update how the PARAM data is handled.
Also, this will be how the fragments interact with the Fog Graphics
Pipeline.
About the next steps:
1) I need to add support for PARAM arrays and vector arrays (arrays of
vectors).
2) I need to add more OPCodes.
3) I need to continue to refactor and make the design cleaner. This will be
ongoing.
Now there are 3 classes, Compiler, Symbol, Context.
The code generation is part of the Symbol - I don't like that. I will
create a Code Generator class.
CodeGenerator is good name, because it will not mess with AsmJit::Compiler
The Symbol class needs to be broken into an interface and child
implementations so that different types of symbols may be supported (TEMP,
PARAM, etc.)
This would be good
4) Now the project just runs the test code -- and is just has a Windows
project file.
I need to make it an actual library, and have a cmake file for it.
Petr Kobalíček
works for me, one test failed, but I'm able to compile it and run (just for info).
Best regards / S pozdravem
Petr Kobalicek
Mike Tajmajer
Hi Petr,
Great!
I haven't had a chance to work on it lately -- I know what I want to do next, but I need to finish up some "paying work".
Sorry!
--------------------------
Mike Tajmajer
iGraphicsGroup.com, Inc.
Notice: This e-mail transmittal is a confidential communication from iGraphicsGroup.com, Inc., and is intended only for the use of the individual addressed. This information may be confidential, privileged or not subject to disclosure under applicable law or regulation. Any dissemination, copying or distribution of this communication is strictly prohibited. If you have received this communication in error, please immediately notify us by return email and delete the message. Thank you for your cooperation.