From all the test I made on the various way to set the normal back in maya, It looks like there is no way to set the normals without locking them.
The only methods available on MFnMesh so set polygon normales are setFaceVertexNormals() and setVertexNormals().
Both functions are considered by Maya as specifying "user" normals. e.i. normals that are by definition locked. In a maya scene, those user normals exists only if one normal is locked (manually), if a normal is set (witht the set vertex normal tool) or if you use the "average normal" function or any normal operation (including transfer attribute and others).
Those normal are set in object space and doesn't change when the mesh is deformed.
When displaying the vertex normal of the mesh, you can see that the normal turns yellow when locked (it's green when not locked).
If no such normals exists on the mesh, maya doesn't even write normal informations when saving the file. It only specify the hardness for every edge (soft or hard).
The correct way to rebuild the normals would be to interpret the alembic normal data to know if an edge is hard or soft, and then to use the setEdgeSmoothing() method.
I made a test with this feature, simply checking if both vertex of each edge have different normal for each adjacent faces. I was ignoring all possible user normals for the test.
It worked pretty well, creating only "green" normals while respecting the original mesh edge hardness. But it's a bit heavy and it takes twice the time to set the normals. There should be place for optimisation though.
I also simply tried to unlock the normal after setting them, but it's very very very slow, on a production scene, the whole normal operation on the file went from 10 sec to 70 sec. And it breaks all data, as unlocking a user normal simply reset it to default (respecting the edge hardness value).
I will probably be creating a new property on my mesh while exporting that would contain a simple boolean array to store the edge hardness of the mesh and another property to store possible information on user normals (alongside the default alembic normal property to allow other software to read the normals correctly).
I assume there should be a more efficient way because opening a mayaAscii file is way faster than opening the same alembic data when reading all normals.
On my test file with only user normals (forcing maya to write them all to the file), the ascii takes 4 seconds to load and the alembic around 12 seconds, but it spends 10 seconds on the normal. And it's precisely 10 seconds spent on MfnMesh::setFaceVertexNormals().
So I guess maya is not using this function or the speed should match (alembic should even be faster than ascii as it is the case when normals are ingored) but there seems to be no other way to set them from the API. I even looked in the MFnMesh.h file to see if there were some hidden functions but without success (I already found a hidden function for uvSet, so ...).
I will continue to try and improve the current normal handling in the alembic file using edge hardness but I would love more information on builtin maya normal handling.
Hans