RFC: pre upload of documents

6162 views
Skip to first unread message

Jeremy Thomerson

unread,
Jan 25, 2011, 6:41:37 PM1/25/11
to bigblueb...@googlegroups.com
Folks,

  I've just uploaded a partial implementation of a new beta feature - pre-uploading of documents for meetings.  I want your feedback.  The idea is simple: third party apps should be able to upload documents in the meeting "create" command so the user can just focus on starting the meeting when he joins.  The implementation is simple: rather than the normal HTTP GET, you send a POST.  The URL is exactly the same as your GET would be (all the normal parameters go there).  The body is just XML (not like an HTML form post).  The XML can contain configuration for multiple modules.  In this case, I'm only handling things for the presentation module, but this gives us the flexibility to allow configuration for different modules later without drastic changes to the XML.


An example of the XML is below.  Please note there are two different ways of uploading files.  The first is to give it a URL to a file.  It will download and process it.  The second is to send a file as a base64 encoded string.

<?xml version="1.0" encoding="UTF-8"?>
<modules>
    <module name="foo">
        <bar something="123" />
    </module>
    <module name="presentation">
        <document url="http://www.samplepdf.com/sample.pdf" />
        <document name="sample-presentation.pdf">JVBERi0xLjQKJ.... [ clipped here ] ....0CiUlRU9GCg==</document>
    </module>
</modules>

I have the uploading (or decoding the base64 files) and processing working.  What isn't working (I'm going to ask Richard or Denis for help) is then telling BBB that the file is done processing so that it automatically displays when the first user shows up.

Please give your feedback.  I'm specifically looking for:
1 - feedback on XML format - remember that we want something generic that can be used for other modules down the road (voice, video, etc...)

2 - other use cases to consider - we definitely won't be getting all of them working right up front, but it's good to know of them so that we can design for them to be added easier later.

Thanks!
Jeremy Thomerson

Richard Alam

unread,
Jan 26, 2011, 10:51:45 AM1/26/11
to bigblueb...@googlegroups.com
Hi Jeremy,

Answer inline.

On Tue, Jan 25, 2011 at 6:41 PM, Jeremy Thomerson
<jer...@thomersonfamily.com> wrote:
> Folks,
>   I've just uploaded a partial implementation of a new beta feature -
> pre-uploading of documents for meetings.  I want your feedback.  The idea is
> simple: third party apps should be able to upload documents in the meeting
> "create" command so the user can just focus on starting the meeting when he
> joins.  The implementation is simple: rather than the normal HTTP GET, you
> send a POST.  The URL is exactly the same as your GET would be (all the
> normal parameters go there).  The body is just XML (not like an HTML form
> post).  The XML can contain configuration for multiple modules.  In this
> case, I'm only handling things for the presentation module, but this gives
> us the flexibility to allow configuration for different modules later
> without drastic changes to the XML.

This is great. We've also been thinking about how to allow 3rd party
apps to pass in configuration to BigBlueButton. Doing it this way will
allow 3rd party apps to tailor what each user's configuration. We'll
call it (for now) dynamic config api as we're also thinking of
allowing dynamic/custom config.xml that gets loaded by the client.
I'll explain this more below.

> The code is in a new branch:
> https://github.com/bigbluebutton/bigbluebutton/tree/preupload_docs_beta
> An example of the XML is below.  Please note there are two different ways of
> uploading files.  The first is to give it a URL to a file.  It will download
> and process it.  The second is to send a file as a base64 encoded string.
> <?xml version="1.0" encoding="UTF-8"?>
> <modules>
>     <module name="foo">
>         <bar something="123" />
>     </module>
>     <module name="presentation">
>         <document url="http://www.samplepdf.com/sample.pdf" />
>         <document name="sample-presentation.pdf">JVBERi0xLjQKJ.... [ clipped
> here ] ....0CiUlRU9GCg==</document>
>     </module>
> </modules>
> I have the uploading (or decoding the base64 files) and processing working.
>  What isn't working (I'm going to ask Richard or Denis for help) is then
> telling BBB that the file is done processing so that it automatically
> displays when the first user shows up.

Currently, BigBlueButton assumes that the presenter is logged-in into
the conference when uploading the presentation. The document
conversion service will send conversion updates to all clients while
processing the uploaded document. When it's done, it will send a
"document converted" message with an xml containing the links to each
slide. When each client receives this message, it stores the
presentation name into it's list of uploaded presentations. It will
also then download all the thumbnails using the link from the xml
message.

However, it your use case, there will be no client listening for the
conversion update messages.

What I'd like to suggest is using a dynamic config.xml passing in the
pre-uploaded presentation.

Currently, the config.xml is a static file in
/var/www/bigbluebutton/client/conf/config.xml. What if we create this
dynamically? We can pass the pre-uploaded presentation under the
presentation module.
When the presentation module sees that there are pre-uploaded
presentations, it will query BigBlueButton web for these
presentations. You can see an example here
(http://pastebin.com/rgCXcxBk) with the top part having the current
config.xml and the bottom part the new config.xml

What do you think?

I'd also like to suggest where to store the pre-uploaded presentation
to address the enhancement request a few days ago about re-using
pre-uploaded presentations on different meetings/classes. I'm thinking
of having pre-uploaded presentations stored in a separate directory
(e.g. /var/bigbluebutton/pre-uploaded-presentations/<uploaded-presentation-id>)
and not under the current meeting directory (i.e.
/var/bigbluebutton/<meeting-id>/.../presentations). The reason for
this is Record And Playback. If the meeting is being recorded, at the
end of the session, we will be collecting all artifacts (presentation,
slides, audio, etc.) of the meeting and store them into an archive.
This way the "Archiving Service" won't remove the original
pre-uploaded presentation.

When a meeting needs to load a pre-uploaded presentation, the call
just passes the "uploaded-presentation-id" and the pre-uploaded
presentation get's copied into the current meeting's presentation
folder.

Looking for feedback.

Richard


> Please give your feedback.  I'm specifically looking for:
> 1 - feedback on XML format - remember that we want something generic that
> can be used for other modules down the road (voice, video, etc...)
> 2 - other use cases to consider - we definitely won't be getting all of them
> working right up front, but it's good to know of them so that we can design
> for them to be added easier later.
> Thanks!
> Jeremy Thomerson
> http://genericconf.com
>

> --
> You received this message because you are subscribed to the Google Groups
> "BigBlueButton-dev" group.
> To post to this group, send email to bigblueb...@googlegroups.com.
> To unsubscribe from this group, send email to
> bigbluebutton-...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/bigbluebutton-dev?hl=en.
>

--
---
BigBlueButton
http://www.bigbluebutton.org
http://code.google.com/p/bigbluebutton

HostBBB.com

unread,
Jan 26, 2011, 9:33:46 PM1/26/11
to BigBlueButton-dev
Jeremy, Richard This looks great, having the ability to dynamically
create the config.xml on create, and client join will provide a lot of
flexibility along with preloaded presentations directory.

I can see enhancing the moodle integration to store moderator, client
configs, and document load/preload list in mysql table for each
defined meeting to take advantage of this.

regards,
Stephen
hostbbb.com
> BigBlueButtonhttp://www.bigbluebutton.orghttp://code.google.com/p/bigbluebutton- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -

Jeremy Thomerson

unread,
Jan 27, 2011, 12:48:49 AM1/27/11
to bigblueb...@googlegroups.com
Richard, some questions inline.

What happens when a user signs in after presentations have been uploaded?  Surely he gets a list of presentations from somewhere?  I'm guessing, anyway.... 

However, it your use case, there will be no client listening for the
conversion update messages.

What I'd like to suggest is using a dynamic config.xml passing in the
pre-uploaded presentation.

Currently, the config.xml is a static file in
/var/www/bigbluebutton/client/conf/config.xml. What if we create this
dynamically? We can pass the pre-uploaded presentation under the
presentation module.
When the presentation module sees that there are pre-uploaded
presentations, it will query BigBlueButton web for these
presentations. You can see an example here
(http://pastebin.com/rgCXcxBk) with the top part having the current
config.xml and the bottom part the new config.xml

What do you think?

I think that's more complex than what it needs to be for this.  Getting the config.xml dynamic is something we want to eventually achieve, but it has more to do with the "join" call - so that modules can be turned on and off per attendee.  This is more on the meeting creation time, the "create" call.  That part is already working.  You want to be able to create a meeting several minutes in advance of everyone joining so that the conversion can be done before everyone is in the room.

So, for this, I think all that is lacking is a way for the client to immediately retrieve a list of uploaded presentations as soon as they join.  I assumed this was already happening....  if I join after presentations have been uploaded, I must somehow get that list of presentations.  If we could make this happen, it should be a lot easier and cleaner to implement - both now and for third parties.

My concern is that with your suggestion above, we complicate the API for those that want to use this feature.  If they want to create a meeting with presentations already in it, they have to send these in the API "create" call, but then they also need to create a way for their third-party app to generate the dynamic config.xml.  BBB shouldn't be generating the dynamic config.xml for per-user configuration of modules - the third-party app should.  Additionally, the dynamic config.xml should be primarily for "per-user" configuration.  This is "per-meeting" configuration. 

I'd also like to suggest where to store the pre-uploaded presentation
to address the enhancement request a few days ago about re-using
pre-uploaded presentations on different meetings/classes. I'm thinking
of having pre-uploaded presentations stored in a separate directory
(e.g. /var/bigbluebutton/pre-uploaded-presentations/<uploaded-presentation-id>)
and not under the current meeting directory (i.e.
/var/bigbluebutton/<meeting-id>/.../presentations). The reason for
this is Record And Playback. If the meeting is being recorded, at the
end of the session, we will be collecting all artifacts (presentation,
slides, audio, etc.) of the meeting and store them into an archive.
This way the "Archiving Service" won't remove the original
pre-uploaded presentation.

I don't think we should worry about that feature at this point at all.  Denis and I both mentioned reasons in that email.  Your approach removes the "stateless" approach that we are taking with BBB.  The server should not store these pre-uploaded presentations after a meeting is over.  If it does, BBB is no longer stateless.  That's not the feature I am trying to implement at all.  All we need is for these presentations to appear exactly like any presentation that was uploaded during a meeting.

When a meeting needs to load a pre-uploaded presentation, the call
just passes the "uploaded-presentation-id" and the pre-uploaded
presentation get's copied into the current meeting's presentation
folder.

See above reason.  That's a road I really don't think we want to go down.  BBB should remain a stateless conferencing engine.  You should be able to power up and down servers in your conferencing cluster without worrying about which ones already have these pre-uploaded presentations.  Think about it as "just-in-time" uploading (like the JIT compiler).

Looking for feedback.

You got it!
 
Richard

Thanks!!
Jeremy Thomerson

Mike Poupore

unread,
Jan 27, 2011, 12:23:18 PM1/27/11
to bigblueb...@googlegroups.com
I agree with Jeremy that this should be done with the API start meeting call and an external XML file.

The API call for creating meetings just needs to have one extra parameter, which is a URL to the file containing the list of presentations and their settings.  It would be up to the external application to generate this file in an acceptible format.

On the server side, when the meeting is created the Presentation service needs to fetch the XML file from this URL, parse it to get the list of presentations, and then download and process each one.  If you include an EMail address in the XML file then it could send a status report when it finished processing, as otherwise there would be very little feedback from this process on unsupported file types, incorrect file names, etc.

On the client side, you can access the parameter from the conference object, open the XML file, get the names of the presentations and load them into the window.  This would only be done when you becomePresenterIfLoneModerator and the list of presentations is empty.

Mike

Richard Alam

unread,
Jan 27, 2011, 1:52:58 PM1/27/11
to bigblueb...@googlegroups.com
Hi,

Agree with you and Jeremy.

What we'll have to figure out is Jeremy's question.

> What happens when a user signs in after presentations have been uploaded?
>  Surely he gets a list of presentations from somewhere?  I'm guessing,
> anyway....

The list of uploaded presentations are stores in the
bigbluebutton-apps/Red5 server.
https://github.com/bigbluebutton/bigbluebutton/blob/master/bigbluebutton-apps/src/main/java/org/bigbluebutton/conference/service/presentation/PresentationRoom.java

The application listens for the "CONVERSION_COMPLETED" (line 96)
message from bigbluebutton-web whose doing the converting. However,
currently, the meeting/room in bigbluebutton-apps gets created when
the first user joins the meeting (i.e. logged into the bigbluebutton
client). So when the pre-uploaded slide conversion is going on,
nothing is listening for these conversion messages in
bigbluebutton-apps.

When a user joins the meeting, the bigbluebutton client will query the
server for the list of uploaded presentations and the state of the
current presentations (line 301 of
https://github.com/bigbluebutton/bigbluebutton/blob/master/bigbluebutton-client/src/org/bigbluebutton/modules/present/business/PresentSOService.as).
However, for the reason mentioned on the above paragraph, it will
return nothing.

So, we need to figure out a way to populate
https://github.com/bigbluebutton/bigbluebutton/blob/master/bigbluebutton-apps/src/main/java/org/bigbluebutton/conference/service/presentation/PresentationRoom.java
with the pre-uploaded presentation when the first user joins the
meeting.

One way, is for the client (when the first user joins) to query
bigbluebutton-web for a list of pre-uploaded presentation (if any) and
then propagate that information to bigbluebutton-apps so that it gets
stored there. When the other users join, their client will just do
what it's currently doing which is query bigbluebutton-apps with a
list of uploaded presentations.

Another way, is for bigbluebutton-web to store this information in
Redis (we are proposing replacing ActiveMQ with Redis -- see email
from Markos). So when the meeting starts in bigbluebutton-apps, it can
query Redis for any pre-uploaded presentation.

Any other ideas?

Richard

--

Steven Schroedl

unread,
Jan 27, 2011, 2:13:52 PM1/27/11
to bigblueb...@googlegroups.com, bigblueb...@googlegroups.com
Having a dropbox for the lesson plans that is readable from BBB would be great. Then the business logic side could do lesson planning and preprocessing of docs to speed load of larger docs.

Steven Schroedl

503 869-0276 [voice]

-- sent from my so called "smart" phone, so please excuse spelling, formatting, or compiler errors

Jeremy Thomerson

unread,
Jan 27, 2011, 11:40:29 PM1/27/11
to bigblueb...@googlegroups.com
On Thu, Jan 27, 2011 at 12:52 PM, Richard Alam <ritz...@gmail.com> wrote:
Hi,

Agree with you and Jeremy.

What we'll have to figure out is Jeremy's question.

> What happens when a user signs in after presentations have been uploaded?
>  Surely he gets a list of presentations from somewhere?  I'm guessing,
> anyway....

The list of uploaded presentations are stores in the
bigbluebutton-apps/Red5 server.
https://github.com/bigbluebutton/bigbluebutton/blob/master/bigbluebutton-apps/src/main/java/org/bigbluebutton/conference/service/presentation/PresentationRoom.java

The application listens for the "CONVERSION_COMPLETED" (line 96)
message from bigbluebutton-web whose doing the converting. However,
currently, the meeting/room in bigbluebutton-apps gets created when
the first user joins the meeting (i.e. logged into the bigbluebutton
client). So when the pre-uploaded slide conversion is going on,
nothing is listening for these conversion messages in
bigbluebutton-apps.

Ah!  That's the piece I was missing.  Red5 hasn't actually created the room yet.
 
When a user joins the meeting, the bigbluebutton client will query the
server for the list of uploaded presentations and the state of the
current presentations (line 301 of
https://github.com/bigbluebutton/bigbluebutton/blob/master/bigbluebutton-client/src/org/bigbluebutton/modules/present/business/PresentSOService.as).
However, for the reason mentioned on the above paragraph, it will
return nothing.

So, we need to figure out a way to populate
https://github.com/bigbluebutton/bigbluebutton/blob/master/bigbluebutton-apps/src/main/java/org/bigbluebutton/conference/service/presentation/PresentationRoom.java
with the pre-uploaded presentation when the first user joins the
meeting.

One way, is for the client (when the first user joins) to query
bigbluebutton-web for a list of pre-uploaded presentation (if any) and
then propagate that information to bigbluebutton-apps so that it gets
stored there. When the other users join, their client will just do
what it's currently doing which is query bigbluebutton-apps with a
list of uploaded presentations.

Another way, is for bigbluebutton-web to store this information in
Redis (we are proposing replacing ActiveMQ with Redis -- see email
from Markos). So when the meeting starts in bigbluebutton-apps, it can
query Redis for any pre-uploaded presentation.

Any other ideas?

I think it could be even simpler.  I haven't looked at the piece of code you mention above that is listening for the "DOCUMENT_CONVERTED" events.... but, I would suspect that it is receiving that, and then looking for the corresponding room that it is associated with, and if it doesn't find that room, discarding the event.  Perhaps instead it could create the previously-uncreated room when it receives an event for an unknown room.  

Thoughts?
 
Jeremy Thomerson

Richard Alam

unread,
Jan 29, 2011, 10:43:01 AM1/29/11
to bigblueb...@googlegroups.com
On Thu, Jan 27, 2011 at 11:40 PM, Jeremy Thomerson
<jer...@thomersonfamily.com> wrote:

<snip/>

> I think it could be even simpler.  I haven't looked at the piece of code you
> mention above that is listening for the "DOCUMENT_CONVERTED" events.... but,
> I would suspect that it is receiving that, and then looking for the
> corresponding room that it is associated with, and if it doesn't find that
> room, discarding the event.  Perhaps instead it could create the
> previously-uncreated room when it receives an event for an unknown room.
> Thoughts?

Unfortunately, we cannot. The life of the room is tied to the presence
of a user in a room. The room gets created when the first user joins
and destroyed when the last user leaves. If the first user joins and
the leaves, the informations gets lost. So when she rejoins, the
pre-uploaded list of presentations are not available.

Richard

>
> Jeremy Thomerson
> http://genericconf.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "BigBlueButton-dev" group.
> To post to this group, send email to bigblueb...@googlegroups.com.
> To unsubscribe from this group, send email to
> bigbluebutton-...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/bigbluebutton-dev?hl=en.
>

--

Mike Poupore

unread,
Jan 30, 2011, 12:42:47 PM1/30/11
to bigblueb...@googlegroups.com
I am still assuming that there will be an XML file generated by the
external application with the list of files to download and any
parameters required.

Presentations are currently uploaded to the folder
{conferenceid}/{roomid}, which is dynamically created when the first
presentation is uploaded. Both parameters are currently set to the
meetingID, but the meetingID does not exist until somebody joins the
room and is no longer relevant after the last person leaves the meeting.

If the external XML file specifies the conferenceid and roomid as
parameters, then the files could be downloaded and converted into that
folder (English101/Week7) with no dependency on whether a meeting has
already started or not. The upload and conversion could start as soon
as the meeting was created with the API, since the meetingID is not
required to determine the name of the folder. This also allows you to
make a call to the API at any time to pre-load a set of documents, even
if nobody ever joins that meeting.

When the client loads, it should also read the XML file to get all the
dynamic parameters for this meeting, which will include the list of
presentations. If the presentations listed in the XML file are not
already in presentationNames[] and you are the moderator, then you can
request that they are loaded using the conferenceID, roomID, and
presentationName listed in the XML file (English 101, Week 7,
Assignment1.pdf). This allows you to load the same presentations over
and over again whether you are in the same meeting or in different
meetings, and does not rely on any data structures on the server.
This would only change the behavior of the client for first person to
enter the room (ie the lone moderator).

As an aside, having the client read an external XML file when it starts
should make customizing the client easier. This allows you to pass any
number of parameters into the client without modifying the server, and
allows you to keep all the business logic off the BBB server and in the
external application where it belongs.

Mike

Reply all
Reply to author
Forward
0 new messages