Long loading time for subforms

[Alex D]

Hi Everyone,

I have an admin view with one large subform:

- 6 columns 

- 30 rows

The field are basic fields like dates and text, but there are 2 sql query fields.

The issue is that the pages load slower the bigger the subform gets. Any ideas to speed this up?

As of right now I can't load some pages anymore.

[Alex Dings]

I found that when removing the sql fields from the subform, the loadspeed improves significantly.

The biggest difference is in the  Access:preloadPermissions (com_mycomponent)  section. 

Without the sql fields

Before Access:preloadPermissions (com_mycomponent) 325ms

With the sql fields

Before Access:preloadPermissions (com_mycomponent) 1179ms

--
You received this message because you are subscribed to the Google Groups "Joomla Component Builder" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jcb+uns...@vdm.io.
To view this discussion on the web visit https://groups.google.com/a/vdm.io/d/msgid/jcb/86d2c720-58b6-43b2-9254-cf9ebbb4fcf8%40vdm.io.

[((ewe))yn]

Hi Alex,

The subforms are completely a Joomla implementation, so the fact that is gets slow when you have many of them in a view, and that the sql fields make them extra slow is all due to how Joomla load the from. JCB basically just builds the XML that gets given to Joomla API that builds the forms.

My approach to this problem was to move all the subforms to there own tables/views as you can see in the admin view in how we link in the fields and the other features related to the admin views. This way we can push the individual subfroms up to a hundred + rows and it still loads... but having many subforms on one page is a huge issue, and why we had to move them out in the first place, when news of the removal of the repeatable fields was made.

Repeatable fields worked verywell, and added little to no load to the page.... why they would not adapter it and keep it in the fields options was very sad to me. So here we are, we only have the subforms to work with and it does come with limitations.

[Alex Dings]

Hi Llewellyn,

Thank you for your prompt and thorough response.

I can understand how this issue is a Joomla issue.

Out of curiosity, do you think I am best off by moving the subforms to other admin views as done in JCB? Or do you think changes will make this easier in the future?

The reason why I ask is that I would like to try to avoid custom things which should normally be covered by normal functionality - in this instance subforms.

Copyright © Vast Development Method | Terms of Service | Privacy Policy | Company

--
You received this message because you are subscribed to the Google Groups "Joomla Component Builder" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jcb+uns...@vdm.io.

To view this discussion on the web visit https://groups.google.com/a/vdm.io/d/msgid/jcb/c86020b3-cfa7-4e41-ab11-20ec0ddb4460%40vdm.io.

[((ewe))yn]

The problem is the amount of fields a browser can load per/view and the amount of values that can be posted per/form, as there are actually browser limitations (from 50+) and server limitations (from 400+). The moment you have more then 50 fields per/view then you are moving in on these limitations (not on all systems) as they are different per/system and browser, again based on memory and default settings and available resources per/computer (visitor) and server (website-host).

So can Joomla solve the problem, I don't think so, unless they move the forms (save and load) over to ajax methods that can speedup the browser and the general load and store of big data sets (like they did in the permissions area of component options).

I don't think moving fields to there own views, and tables are custom solutions, but a basic and simple solution that speedup the user experience.

I am building the member manager, that will server to demonstrate linked components, not only linked views. So I am also down sizing the component number of views, and instead moving views to their own components, and then linking them in to one main component with various conventions and helper methods. Basically pushing the development to smaller components that intelligently interact and merge forms, data sets and results for easy scalability, that is more manageable in Joomla.

Subforms are nice tools, but should not become a shortcut that produce slow page loads and as result bad user experience. This one principle always helps, and that is can it scale well, if not look for another path within good convention.

So it comes down to the target audience of your component, what are the lowest limitation they will have in relation to the above mentioned realities. That is what holds us back to using more freedom of design and rapid development.

Lastly I am not that aware of the changes happening on the Joomla side of things, and it may be that other are also debating the progress on this front.... so if you have time search the Joomla forms and see if you can find something on the subject, and if not... well you can always start the subject. I would also be interested on how they think this should be resolved... just do not have the time right now to take on this research. But since you are looking for a better answer go for it :)

[((ewe))yn]

By the way, the reason the old repeatable fields worked better is that it converted the fields into one input field as a json string that was posted to the server as one value/input field and not as a per/field post and/or load, which is the way the subform works.

[Alex D]

Just to be sure that you don't think I ignored your last answer. I am working on a couple of things you mentioned to see which works best for the users of the component.

I will update this thread with the final result.

[((ewe))yn]

Little bit of news on this front, since we added the option to add custom dabs to Admin views, I have build a little script that pulls data from other database, and load the fields in one of these custom tabs, and then on save, takes those values again and update the other database. Neat little trick :) here is the code:

For the load of the custom tab, I know it has stuff that you may not know what it is doing, but try to get the general flow idea:

/**

 * Load the tabs

 *

 * @param   object   $item     Data for the form

 * @param   string   $view     The view name

 * @param   string   $return   The return value if found

 *

 * @return string

 *

 */

public static function loadDynamicTabs(&$item, $view = 'member', $return = '')

{

// only loads if type and account is set

if (is_numeric($item->type) && $item->type > 0 && is_numeric($item->account) && $item->account > 0)

{

// get all the available component calling metods

$class = new ReflectionClass('[[[Component]]]Helper');

$methods = array_filter($class->getMethods(ReflectionMethod::IS_PUBLIC),

function ($method) {

if (strpos($method->name, 'get') !== false && strpos($method->name, 'Availlable') !== false ) // The spelling mistake (Availlable) is to unique identify those classes.

{

return true;

}

return false;

}

);

// set the tabs

$tabs = array();

$layout = array();

if (self::checkArray($methods))

{

foreach ($methods as $method)

{

// get components

$components = self::{$method->name}($item->type, $item->account);

// check if we found components

if (self::checkArray($components))

{

// get assessment details

foreach ($components as $_name => $component)

{

if (self::checkArray($component))

{

$tables = array();

foreach ($component as $_nr => $comp)

{

if (($ids = self::getVars('form', $item->id, $view, 'id', 'IN', str_replace('com_', '', $comp->element))) !== false && self::checkArray($ids))

{

$tables[] = self::getTabLinksTable($ids, $item, $comp, $view, $return);

}

}

// load the tables to the layout

if (self::checkArray($tables))

{

foreach ($tables as $table)

{

if (self::checkString($table))

{

if (!isset($layout[$_name]))

{

$layout[$_name] = $table;

}

else

{

$layout[$_name] .= $table;

}

}

}

}

// add layout to tabs

if (self::checkArray($layout) && count((array) $layout) == 2)

{

$tabs[] = self::setTab($layout, $view);

$layout = array();

}

}

elseif (self::checkObject($component) && isset($component->element))

{

if (($id = self::getVar('form', $item->id, $view, 'id', '=', str_replace('com_', '', $component->element))) === false) // get item ID

{

// if no item was found set to zero

$id = 0;

}

// check if user are allowed to edit form values or create form values

if (($id > 0 && JFactory::getUser()->authorise('form.edit', $component->element . '.form.' . (int) $id)) || ($id == 0 && JFactory::getUser()->authorise('form.create', $component->element)))

{

$fields = self::getTabFields($id, $component);

// load the fields to the layout

if (self::checkString($fields))

{

$layout[$_name] = $fields;

}

// add layout to tabs

if (self::checkArray($layout) && count((array) $layout) == 2)

{

$tabs[] = self::setTab($layout, $view);

$layout = array();

}

}

}

}

}

}

}

// add layout to tabs

if (self::checkArray($layout))

{

$tabs[] = self::setTab($layout, $view);

}

// check if we have tabs

if (self::checkArray($tabs))

{

return implode("\n", $tabs);

}

}

return '';

}

/**

 * get the tabe fields

 *

 * @param   int      $id          The item id

 * @param   object   $component   The target component details

 *

 * @return string

 *

 */

protected static function getTabFields($id, &$component)

{

// build the rows

$rows = '';

// get the form

if (method_exists(__CLASS__, "getMemberForms") && ($form = self::getMemberForms($id, $component->element)) !== false && self::checkObject($form))

{

// get the fields for this form

if (($fields = JComponentHelper::getParams($component->element)->get('edit_fields', false)) !== false && self::checkObject($fields))

{

// add the id field if the id was found (but hidden)

if ($id > 0)

{

$form->setFieldAttribute('id', 'type', 'hidden');

$rows = $form->renderField('id');

}

// add the rest of the fields

foreach ($fields as $row)

{

if ($form->getField($row->field))

{

$rows .= PHP_EOL . $form->renderField($row->field);

}

}

}

}

return $rows;

}

/**

 * get the tab table of links

 *

 * @param   array    $ids         The target ids

 * @param   object   $item        The target item details

 * @param   object   $component   The target component details

 * @param   string   $view        The view name

 * @param   string   $return     The return value if found

 *

 * @return string

 *

 */

protected static function getTabLinksTable($ids, &$item, &$comp, &$view, &$return)

{

// set some defaults

$_return = '&ref=' . $view . '&refid=' . $item->id . '&return=' . urlencode(base64_encode('index.php?option=com_[[[component]]]&view=' . $view . '&layout=edit&id=' . $item->id . $return));

$rows = array();

// add a row to create a new item

if (($create_button = self::getCreateButton('form', 'forms', $_return, $comp->element)) !== false && self::checkString($create_button))

{

$rows[] = '<td data-column="'.$comp->name.'">' . $create_button . '</td>';

}

// build the links

foreach ($ids as $id)

{

$created = self::getVar('form', $id, 'id', 'created', '=', str_replace('com_', '', $comp->element));

$rows[] = '<td data-column="'.$comp->name.'">' . self::fancyDayTimeDate($created) . self::getEditButton($id, 'form', 'forms', $_return, $comp->element) . '</td>';

}

// set the header

$head = array($comp->name);

// return the table

return self::setSubformTable($head, $rows, $view . '_' . $comp->name);

}

then to store the values again:

/**

 * save the dynamic values

 *

 * @param   object   $date     The main Data

 * @param   string   $view     The view name

 *

 * @return string

 *

 */

public static function saveDynamicValues(&$data, $view = 'member')

{

// get all the available component calling metods

$class = new ReflectionClass('[[[Component]]]Helper');

$methods = array_filter($class->getMethods(ReflectionMethod::IS_PUBLIC),

function ($method) {

if (strpos($method->name, 'get') !== false && strpos($method->name, 'Availlable') !== false )

{

return true;

}

return false;

}

);

// check if we have methods

if (self::checkArray($methods))

{

// get the app object

$app = JFactory::getApplication();

// get the post object

$post = JFactory::getApplication()->input->post;

// get the user object

$user = JFactory::getUser();

// get the database object

$db = JFactory::getDBO();

// start looping the metods

foreach ($methods as $method)

{

// get components

$components = self::{$method->name}($data['type'], $data['account']);

// check if we found components

if (self::checkArray($components))

{

// get assessment details

foreach ($components as $_name => $comp)

{

// only save one to one components

if (self::checkObject($comp) && isset($comp->element))

{

$_component = $comp->element;

$component = str_replace('com_', '', $_component);

$Component = self::safeString($component, 'F');

$COMponent = self::safeString($component, 'W');

// get the posted date if there were any

$_data  = $post->get($component, array(), 'array');

// check if user are allowed to edit form values or create form values

if (self::checkArray($_data))

{

// make sure the ID is set

if (!isset($_data['id']) || !is_numeric($_data['id']))

{

$_data['id'] = 0;

}

// check if user may edit

if ($_data['id'] > 0 && !$user->authorise('form.edit', $_component . '.form.' . (int) $_data['id']))

{

// check edit own

if (($created_by = self::getVar('form', $_data['id'], 'id', 'created_by', '=', $component)) === false || $created_by != $user->id || !$user->authorise('form.edit.own', $_component))

{

$app->enqueueMessage(JText::sprintf('You do not have permission to edit %s, please contact your system administrator.', $COMponent, $_data['id']), 'warning');

continue;

}

}

// check if user may create

if ($_data['id'] == 0 && !$user->authorise('form.create', $_component))

{

$app->enqueueMessage(JText::sprintf('You do not have permission to add data to %s, please contact your system administrator.', $COMponent), 'warning');

continue;

}

// make sure the member ID is set if view is member

if ('member' === $view && !isset($_data[$view]) || !is_numeric($_data[$view]) || $_data[$view] == 0)

{

if ($_data['id'] > 0 && $data['id'] > 0)

{

// get the member ID

if (($member = self::getVar('form', $_data['id'], 'id', $view, '=', $component)) === false || $member != $data['id'])

{

$app->enqueueMessage(JText::sprintf('Member ID mismatch, %s-%s could not be saved.', $COMponent, $_data['id']), 'error');

continue;

}

}

elseif ($_data['id'] > 0)

{

// get the member ID

if (($member = self::getVar('form', $_data['id'], 'id', $view, '=', $component)) === false || $member == 0)

{

$app->enqueueMessage(JText::sprintf('Member ID mismatch, %s-%s could not be saved.', $COMponent, $_data['id']), 'error');

continue;

}

}

elseif ($data['id'] > 0)

{

// get the member ID

$member = $data['id'];

}

else

{

$app->enqueueMessage(JText::sprintf('%s could not be saved, please try again. This could be due to the fact this the member ID was not ready.', $COMponent), 'error');

continue;

}

// set the member ID

$_data[$view] = $member;

}

// get the model

$model = self::getModel('form', JPATH_ADMINISTRATOR . '/components/' . $_component, $Component);

// do we have the model

if ($model)

{

// force other component path (TODO) will be an issue if forms and fields are the same

\JForm::addFormPath(JPATH_ADMINISTRATOR . '/components/' . $_component . '/models/forms');

\JForm::addFieldPath(JPATH_ADMINISTRATOR . '/components/' . $_component . '/models/fields');

// Validate the posted data.

// Sometimes the form needs some posted data, such as for plugins and modules.

$form = $model->getForm($_data, false);

if (!$form)

{

$app->enqueueMessage($model->getError(), 'error');

continue;

}

// remove all fields not part of the allowed edit fields

if (($fields = JComponentHelper::getParams($_component)->get('edit_fields', false)) !== false && self::checkObject($fields))

{

// build a fields array bucket

$fieldActive = array();

foreach ($fields as $row)

{

$fieldActive[$row->field] = $row->field;

}

// set the keep values

$fieldActive['id'] = 'id';

$fieldActive['member'] = 'member';

$fieldActive['asset_id'] = 'asset_id';

$fieldActive['created'] = 'created';

$fieldActive['created_by'] = 'created_by';

$fieldActive['modified'] = 'modified';

$fieldActive['modified_by'] = 'modified_by';

$fieldActive['version'] = 'version';

$fieldActive['rules'] = 'rules';

// get the database columns of this table

$columns = $db->getTableColumns("#__" . $component . "_form", false);

// no make sure the fields that are not editable are removed (so can't be updated via this form)

foreach(array_keys($columns) as $field)

{

if (!isset($fieldActive[$field]))

{

$form->removeField($field);

}

}

}

// Send an object which can be modified through the plugin event

$objData = (object) $_data;

$app->triggerEvent(

'onContentNormaliseRequestData',

array($_component . '.form', $objData, $form)

);

$_data = (array) $objData;

// Test whether the data is valid.

$validData = $model->validate($form, $_data);

// Check for validation errors.

if ($validData === false)

{

// Get the validation messages.

$errors = $model->getErrors();

// Push up to three validation messages out to the user.

for ($i = 0, $n = count($errors); $i < $n && $i < 3; $i++)

{

if ($errors[$i] instanceof \Exception)

{

$app->enqueueMessage($errors[$i]->getMessage(), 'warning');

}

else

{

$app->enqueueMessage($errors[$i], 'warning');

}

}

continue;

}

// Attempt to save the data.

if (!$model->save($validData))

{

$app->enqueueMessage(JText::sprintf('%s data could not be saved', $COMponent), 'error');

}

}

}

}

}

}

}

}

}

I will soon be pushing out a member manager component that will server as a foundation to scale member related form data relations in separate components due to dynamic relationships. I am going to use it as the foundation of a patient evaluation manger. I am very exited about this... of course the base component will be opensource on github (and maybe one or two addon component to demonstrate the relationships structure), but the extending component will be payed options.

This above code is from that project, and I think it is going to take JCB in to huge scale able projects. I mean, here is the idea, you know how JCB can export components and import them seamlessly into another JCB... well what if that feature becomes available to all components build with JCB, and we can build a bridge between servers that maintain synchronization of Joomla websites across multiple Joomla instances. This is the future of JCB and where I am planning to park a subscription list of features.

Now we can have a client/member manager with multiple components dealing with many different kind of data and scale right inside Joomla. I know this is little of topic... but dealing with many fields on a page is sort of resolved in this approach, well at least for me.

[Alex D]

Hi guys,

I have resolved my loading issue by building custom list/SQL fields that on page load are:

1. Empty / selected option
2. Basic, have no chosen.js applied (data-chosen="true")
   

Then on click:

1. Apply chosen.js
   
2. Trigger chosen.js open

Then on showing_dropdown:

1. Get previously selected item
2. Fill list with items
3. Select the previously selected item
4. Update chosen.js

Then on hiding_dropdown:

1. Remove all items which are not selected
2. Destroy chosen.js

This solves the loading issue chosen.js puts on the page and the strain on the DOM of having 1000s of options per list field (times 200).

[((ewe))yn]

Care to share the code?