5.4 Release Candidate

[Nic Pottier]

Hi all,

We are pleased to announce that we have pushed the first release candidate for the 5.4 release (v5.4.3) to the https://github.com/rapidpro/rapidpro repo. We consider this release stable but are entering a bug fixing stage over the next week and will fix any critical bugs found during the next two weeks. Expect a final announcement of the official stable 5.4 release on March 16th as well as what we are planning for the 6.0 release due in July.

As always, this release wouldn’t be possible without the continued support of Nyaruka’s customers at UNICEF and TextIt (https://textit.in). A big thanks from the Nyaruka team for choosing us for your RapidPro hosting, the continued development of RapidPro wouldn’t be possible without it. Big thanks to Eric, Leah, Norbert and Rowan for their hard work in putting this release together.

We also received some valuable pull requests from Ilhasoft (globals, VK channel) and Resistbot (globals, freshchat, archiver). As always, we really appreciate the collaboration!

Big updates in this release

• We’ve introduced a context explorer that you can use during simulation. Not only does this show you all the variables in the context that you can access in your flow, but it updates dynamically as you walk through the flow, letting you see how things change. You can also copy the expression to your clipboard for use in an expression with a single click. Look for the @ symbol on the simulator to open the context explorer.
• When starting a subflow the editor now lets you specifically set variables that are referenced in the subflow. This makes using flows as “functions” much easier and more natural. Just look for the “parameters” tab in the “Start another flow” action.
• We’ve added the concept of “Issues” on flows. These may be missing dependencies, errors in expressions or other errors that you should address. Nodes that have issues are highlighted in red and an explanation is given as to what the problem is. This was a big step for us to move towards having “soft” dependencies, which make maintaining an account a bit easier. We will continue to add inspections and issues as time goes on but we are really excited to have this new tool to help highlight possible problems in people’s flows.
• In cooperation with Ilhasoft and Resistbot we added the concept of “Flow Globals”. These are account levels globals that you can access in your flows. Useful for storing things such as API tokens or “site-wide” constants for use across flows.
• We’ve added the concept of Facebook “Tags” or “Topics” for outgoing messages. See here for more: https://developers.facebook.com/docs/messenger-platform/send-messages/message-tags/ This allows accounts to send outbound messages while satisfying Facebook tagging policies.
• You can now restrict tel channels to only send within the same country to prevent any international charges in the case of misconfiguration. Look for the checkbox on your channel configuration to change this setting.
• We expanded our search syntax to allow searching by group membership. For example you can have a search like group = Educators and age > 30
• You can now pick the level of detail for events on the contact history page by selecting either the “Full” or “Summary” view on the contact page. This lets you see a more message focused view as we had before or a detailed view showing all flow events.
• We changed our handling of webhooks to make them more useful. They automatically retry when appropriate and we have increased the max payload size to 1 megabyte so that larger payloads can be used. Note that with that change, large webhook bodies are only available in the sprint they are executed on as we do not store those giant webhook responses in the session.
• We have removed the option of creating new flows in the old editor, note that we will be removing the old editor entirely for 6.0. (old flows will be automatically migrated to the new editor)
• We’ve submitted a Community Datasource for Google Data Studio which will let you import results from RapidPro flows for use within Google Data Studio. More details on that once that is approved by the Google Data Studio folks.

Other important work:

• We’ve been continuing to move towards using web-components internally for RapidPro dialogs and other rich widgets. We have moved to using those components on the campaign, messaging and flow start dialogs.
• We moved most search processing over to Mailroom in an effort to unify the places that need to understand and evaluate our contact query language. We still have a bit of work to completely remove the last few pieces of Python based ContactQL evaluation but we are well positioned to do that for 6.0
• We added two new channel types, the VK social channel (thanks Ihlasoft!) and the FreshChat channel (thanks ResistBot!)

Commit Tracking

While the .4 release cycle is a bit shorter than the rest due to holidays that didn’t keep the commit counts down, here’s the commit counts across the repos over the past 4 months:

• 830 RapidPro commits
• 316 GoFlow commits
• 230 Mailroom commits
• 169 FlowEditor commits
• 81 Courier commits
• 16 Archiver commits
• 0 Indexer commits

Component Changes

RapidPro Improvements

• Add URNs to runs API endpoint
• Add tabs to toggle between full flow event history and summary of messages
• Allow ability to restrict channels to not send internationally
• Use mailroom searching for Omnibox results
• Dynamic group creation / reevaluation through Mailroom
• Use Mailroom for contact searches, contact list pages and flow starts via search, evaluate dynamic groups in Mailroom
• Warn users if starting for Facebook without a topic
• Optimize API explorer to work with large result payloads
• Support the WhatsApp templates HEADER and FOOTER components
• Convert send message dialog to components
• Convert add/edit campaign event to components
• Add UI for managing globals
• Add error and failure events to contact history
• Add FreshChat channel support (thanks Resistbot!)
• Add VK Channel (thanks @Ihlasoft)
• Allow on demand syncing of classifiers

Editor Improvements

• Add context explorer during simulation
• Add issues tab, highlight issues in flows
• Add subflow parameters to editor
• Add Facebook topic selection
• Make editor completion case insensitive
• Localization to Portuguese (thanks Ilhasoft for inspiration / translation!)
• Add Split by Scheme

GoFlow Engine Improvements

• Add trim, trim_left and trim_right expression functions
• Add issues to report on invalid regexes, missing dependencies during flow saving
• Quote telephone numbers when rewriting contact queries
• Do not assign international messages to channels that are not configured to send internationally
• Interpret contact queries which are formatted phone numbers as tel = queries
• Allow querying for contacts using urn keyword to match any urn type
• Allow querying for contacts using group to match any group name
• Automatically retry HTTP web hooks (thanks Resistbot for inspiration!)
• Loosen email regex used by has_email test
• Add @globals to hold org-level constants
• Add topic to send_msg actions to allow selection of Facebook tags

Mailroom Improvements

• Add new endpoint to search / parse contact queries
• Use mailroom to evaluate / reevaluate dynamic groups
• Optimize asset caching
• Increase max web hook response to 1MB
• Make webhook timeouts configurable

Courier Improvements

• Add VK Channel (thanks Ilhasoft!)
• Allow configuring and sending of quick replies for external channels (thanks Ilhasoft!)
• Change Arabia Cell max length to 670,
• Add support for Twilio Whatsapp channel type
• Set Facebook message type tag when topic is set on message
• Send WhatsApp media via URL
• Ignore status updates for incoming messages

Cheers,

Nic