It is not too simple. Given:
db.define_table('table1', Field('name'))
db.define_table('table2',Field('name'),Field(table1','reference table1'))
define:
def require_unique(form):
then when processing forms apply the above check:
form = SQLFORM(db.table2)
form.process(onvalidation=require_unique)