Performances with huge metadata set

[Emmanuel Dreyfus]

Hello

I am trying to include metadata from eduGAIN. I moved from flat file
to PDO so that the server does not have to parse a huge PHP file on
each request, however I still face performances problems.

I had a look to SimpleSAML\Metadata\MetaDataStorageHandler and if
I understand correctly, looking up a single SP cause the whole set
to be loaded into memory:

For instance SimpleSAML\Metadata\MetaDataStorageSource::lookupIndexFromEntityId
could do an efficient SELECT entity_data FROM $set WHERE entity_id='$entityId'
but instead it just fetches the whole set and interate in it. And since
SimpleSAML\Metadata_MetaDataStorageHandlerPdo does not override it, we
are doomed to load the whole set in memory.

Am I correct? What is the appropriate path to performance?

--
Emmanuel Dreyfus
ma...@netbsd.org

[Patrick Radtke]

Your evaluation looks correct, though I think the method you want to override is getMetaData since lookupIndexFromEntityId is private.

If you use 'serialize' as the format instead of 'flatfile' then each entityID ends up in its own file and avoids the overhead of re-parsing the file.

- Patrick

[Emmanuel Dreyfus]

Patrick Radtke <pra...@gmail.com> wrote:

> Your evaluation looks correct, though I think the method you want to
> override is getMetaData since lookupIndexFromEntityId is private.

Here is a patch that implements the missing method. Any chance it gets merged?

Also: is there any merit to implement an efficient getEntityIdFromHostPath()?

--- lib/SimpleSAML/Metadata/MetaDataStorageHandlerPdo.php.orig
+++ lib/SimpleSAML/Metadata/MetaDataStorageHandlerPdo.php
@@ -141,8 +141,50 @@
$this->cachedMetadata[$set] = $metadataSet;
return $metadataSet;
}

+ /**
+ * Retrieve a metadata entry.
+ *
+ * @param string $entityId The entityId we are looking up.
+ * @param string $set The set we are looking for metadata in.
+ *
+ * @return array An associative array with metadata for the given entity, or NULL if we are unable to
+ * locate the entity.
+ */
+ public function getMetaData($entityId, $set)
+ {
+ assert('is_string($entityId)');
+ assert('is_string($set)');
+
+ $tableName = $this->getTableName($set);
+
+ if (!in_array($set, $this->supportedSets, true)) {
+ return null;
+ }
+
+ $stmt = $this->db->read("SELECT entity_id, entity_data FROM $tableName WHERE entity_id=:entityId", array('entityId' => $entityId));
+ if ($stmt->execute()) {
+ $rowCount = 0;
+
+ while ($d = $stmt->fetch()) {
+ if (++$rowCount > 1) {
+ SimpleSAML\Logger::warning("Dulicate match for $entityId in set $set");
+ break;
+ }
+ $data = json_decode($d['entity_data'], true);
+ if ($data === null) {
+ throw new SimpleSAML_Error_Exception("Cannot decode metadata for entity '${d['entity_id']}'");
+ }
+ if (!array_key_exists('entityid', $data)) {
+ $data['entityid'] = $d['entity_id'];
+ }
+ }
+ return $data;
+ } else {
+ throw new Exception('PDO metadata handler: Database error: '.var_export($this->db->getLastError(), true));
+ }
+ }

private function generateDynamicHostedEntityID($set)
{
assert('is_string($set)');




--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
ma...@netbsd.org

[Jaime Perez Crespo]

Hi Emmanuel,

On 26 May 2018, at 05:05 AM, Emmanuel Dreyfus <ma...@netbsd.org> wrote:
> Here is a patch that implements the missing method. Any chance it gets merged?

Would you mind opening a pull request with your patch? That way we can easily review it, and merge it directly after approved.

> --
> This is a mailing list for users of SimpleSAMLphp, not a support service. If you are willing to buy commercial support, please take a look here:
>
> https://simplesamlphp.org/support
>
> Before sending your question, make sure it is related to SimpleSAMLphp, and not your web server's configuration or any other third-party software. This mailing list cannot help with software that uses SimpleSAMLphp, only regarding SimpleSAMLphp itself.
>
> Make sure to read the documentation:
>
> https://simplesamlphp.org/docs/stable/
>
> If you have an issue with SimpleSAMLphp that you cannot resolve and reading the documentation doesn't help, you are more than welcome to ask here for help. Subscribe to the list and send an email with your question. However, you will be expected to comply with some minimum, common sense standards in your questions. Please read this carefully:
>
> http://catb.org/~esr/faqs/smart-questions.html
> ---
> You received this message because you are subscribed to the Google Groups "SimpleSAMLphp" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to simplesamlph...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

—
Jaime Pérez
Uninett / Feide

jaime...@uninett.no
jaime...@protonmail.com
9A08 EA20 E062 70B4 616B 43E3 562A FE3A 6293 62C2

"Two roads diverged in a wood, and I, I took the one less traveled by, and that has made all the difference."
- Robert Frost