RapidPro 6.2 Release

[Nic Pottier]

Hi all,

Per our usual release schedule we just cut the final 6.2 release of RapidPro and associated components. As always you can find the version numbers to use for this release in the RapidPro README:

https://github.com/rapidpro/rapidpro/blob/main/README.md

This release is mostly setting the groundwork for a big new feature for RapidPro: native ticketing support. We have seen greater and greater demand to be able to do better human agent handoff within RapidPro and after testing the waters with ZenDesk and RocketChat integration we decided that for many use cases a built in ticketer would really simplify things for users. So we are laying the groundwork for that but don’t have anything to show yet for 6.2, expect a first beta in 6.4.

However we’ve still been busy in this release and have lots of other goodies to offer:

• We’ve continued making improvements and bug fixes to the new user interface launched in 6.0 and things are looking better than ever in that respect.
• RapidPro got a full pass from a penetration testing company and we’re happy to report no serious issues were found. We did make some improvements here and there though and those are included in this release.
• We’ve finished up the 2 factor support, so you can now secure logins with an OTP token.
• Per usual after a major release, we squashed our migrations, make sure you install and run migrations on 6.0.8 before running this release.
• You can now forward an ongoing IVR call, say to connect someone to a representative or to create a “switchboard” with extensions.
• We’ve moved a few fields over from 32 bit ints to 64 bit ints, namely msg ids and channel log ids. These can be time consuming migrations, so we recommend testing this with a clone of your database as you may need to incur some downtime when running them. Note that this also means that our main RapidPro install has now sent and received over 2 billion messages. (!!)
• We’ve added the concept of “Background Flows”. This is a new flow type which can be run without interrupting contacts active in other flows. This addresses a use case we’ve been seeing more of: using a flow to massage contact data. You can now do this with a background flow without fear of mucking about with the flow state of contacts currently using your system.
• We’ve updated RapidPro to use ElasticSearch 7 and Postgres 11. (our tests are now running on PG 11 and PG 12)
• Lots of improvements were made to the invite/accept functionality for new users on a workspace.
• Tons of other bug fixes and improvements across all components, we’ll cover all of those when we finalize the release.

As always, the Nyaruka team thanks UNICEF and all our TextIt customers for their ongoing support in allowing us to continue to improve the platform. Please open an issue if you have any questions or run into any issues with this release.

Big big thanks to Eric, Leah, Norbert and Rowan for all their hard work on this release.

Detailed Release Notes:

RapidPro

• Update Portuguese translation
• Update Spanish translation
• Disable old Zenvia channel type
• Add spacing between individual channel log events
• Fix external channel claim form
• Update translations for es, fr and pt-BR
• Fix resthook formax validation
• Require recent authentication to view backup tokens
• Disable syncing of archived Android channels
• Add view to list all possible channel types
• Fix rendering of nameless channels
• Do not allow duplicates invites
• Never respond with registration commands in sync handler
• Enforce time limit between login and two factor verification
• Prevent inviting existing users
• Create failed login records when users enter incorrect backup tokens too many times
• Logout user to force login to accept invite and require invite email account exactly
• Add new 2FA management views
• Use Twilio API to determine capabilities of new Twilio channels
• Fix result pages not loading for users using Spanish interface
• Remove no longer used permissions
• Reduce recent export window to 4 hours
• Change message campaign events to use background flows
• Remove UserSettings.tel and add UserSettings.last_auth_on
• Increase max len of URN fields on airtime transfers
• Add toggle to display manual flow starts only
• Make validation of external channel URLs disallow private and link local hosts
• Cleanup middleware used to set org, timezone and language
• Switch to microsecond accuracy timestamps
• Switch to default_storage for export assets
• Mute invalid host errors
• Add migration to alter m2ms to use bigints
• Drop no longer used database function
• Switch to big id for msgs and channel logs
• Remove unused context processor and unused code from org_perms
• Rework contact history so that rendering as events happens in view and we also expose a JSON version
• Upgrade urllib3
• Update to elastic search v7
• Broadcast events in history should be white like message events
• Add index on flow start by start type
• Allow only deleting msg folders without active children labels
• Use engine events (with some extra properties) for msgs in contact history
• Fix API serialization of background flow type
• Allow background flows to be used in scheduled triggers
• Configure editor and components to use completions files in current language
• Fix deleting classifiers
• Add support for background flows
• Fix importing dependencies when it’s a clone in the same workspace
• Allow aliases to be reused on boundaries with different parent
• Increase max length on external channels to be configurable up to 6400 chars
• Fix contact export warning for existing export
• Add non-empty constraint for URN scheme and path
• Fix contact list pagination with searches
• Show query on list page for smart groups
• Change template translations to be TEXT
• Set global email timeout
• Update tel parsing to match gocommon, fixing how we currently accept local US numbers
• Un-beta-gate Rocket.Chat channels
• Login summary on org home page should include agents
• Rework manage accounts UI to include agents
• Fix deleted flow dependency preventing global deletion
• Cache lookups of auth.Group instances
• For field columns in imports, only match against user fields
• Add agent role and cleanup code around org roles
• Wire table listeners on pjax reload
• Inner scrolling on contact list page
• Improve styles for recipient lists
• Trim our start runs 1,000 at a time and by id
• Increase global max value length to 10,000 and fix UI to be more consistent with fields
• Share modals on globals list, truncate values
• Add security settings file
• Fix intent selection on split by intent
• Use Django password validators
• Fix contact exports when orgs have orphaned URNs in schemes they don’t currently use
• Hide editor language dialog blurb until needed to prevent flashing
• Fix broken flows list page if org has no flows
• Allow underscores in global names
• Improve calculating of URN columns for exports so tests don’t break every time we add new URN schemes
• Make instruction lists on channel claim pages more consistent

Mailroom

• Update to latest goflow with completed es and pt-BR translations
• use SaveRequest() so our ivr logs always have bodies
• Fix campaign event firing for background flows
• IVR forwards for Nexmo and Twilio
• Match referral trigger without case
• Add link local ips to default disallowed networks config
• Update phonenumbers lib
• Decrease locations cache timeout to 1 minute
• Support ElasticSearch 7.2 (backwards incompatible to Elastic 6.*)
• Simplify FCM client code
• Fix updating message status when id column is bigint
• Ensure courier messages are always queued for a single contact
• Fix not triggering FCM syncs for broadcasts and ticket reply messages
• Add support for background flows
• Variable timeout for fire campaign task
• Fix misreporting created contacts as updated during imports
• Ensure field and group assets used for imports are fresh
• Add support for internal type ticketers
• Configure engine to disallow HTTP requests to private networks

FlowEditor

* Update pt-BR translations

• Update Spanish translations
• Fix localization of Quick Replies tab and localization attributes
  Fix editor language detection and add Spanish (es) translation
  Implement dial router
• Update components to version with nicer select expression selection
• Don’t blow up on save if we encounter a server error
  Switch to newer testing framework
  Allow expressions in label form
• Fix open ticket and set channel actions saving full asset to definition instead of reference
  * Fix translation of flow types from API to include background flows
• Allow online actions to be used in background flows
  Fix empty nodes after non-router nodes defaulting to waits in background flows
  Use initial result name on split by intent
  * Add support for background flows
• Fix flow filter
  Use engine flow type names
  Don’t advance cursor when doing gsm replacement

Indexer

• Change to use ElasticSearch v7

Courier

• Add handling for button whatsapp message type
• Add Kaleyra channel type
• switch id to bigserial on messages
• Cache media upload failures localy for 15m
• include header when sanitizing request/response
• Cleanup of whatsapp media handling
• Detect media type for uploading media
• Better logging of error cases when uploading WhatsApp media
• use url.parse to build media URL
• Check and log errors when building URLs for sending