Skip to content

OAE Upgrade Guide

Jump to bottom Edit New page
Salla Karppinen edited this page Jul 6, 2017 · 70 revisions

Table of Contents

  1. Upgrading from OAE 12.6 to OAE 13.0
  2. Upgrading from OAE 12.5 to OAE 12.6
  3. Upgrading from OAE 12.1 to 12.2
  4. Upgrading from OAE 11.3 to 12.0
  5. Upgrading from OAE 11.2 to 11.3
  6. Upgrading from OAE 11.0 to 11.1
  7. Upgrading Elasticsearch from 1.1.1 to 1.5.2
  8. Upgrading from OAE 10.2 to 10.3
  9. Upgrading from OAE 10.1 to 10.2
  10. Upgrading from OAE 10.0 to 10.1
  11. Upgrading from OAE 9.2 to 10.0
  12. Upgrading from OAE 9.1 to 9.2
  13. Upgrading from OAE 8.0 to 9.0
  14. Upgrading from OAE 8.0 to 8.0.2
  15. Upgrading from OAE 7.2 to OAE 8.0
  16. Upgrading from OAE 7.1 to OAE 7.2
  17. Upgrading from OAE 6.5 to OAE 7.0
  18. Upgrading from OAE 6.4 to OAE 6.5
  19. Upgrading from OAE 5.0 to OAE 6.0
  20. Upgrading from OAE 4.4 to OAE 5.0
  21. Upgrading from OAE 4.3 to OAE 4.4

Upgrading from OAE 12.6 to OAE 13.0

OAE 13.0 adds a column secret into the AuthenticationLoginId keyspace for password recovery. This requires the following migration steps:

  1. Upgrade to 13.0

  2. Use cqlsh to add the secret column by running: cqlsh -k oae -f etc/migration/12.6.0-13.0.0/addSecretColumnToAuthenticationLoginId.cql

NOTE: If you are planning on using Jitsi, please update your Nginx config to the latest version in puppet-hilary or, if you are doing the configuration manually, make sure you change the line rewrite ^/me(.*) / permanent; to rewrite ^/me/(.*) / permanent;

Upgrading from OAE 12.5 to OAE 12.6

OAE 12.6 changes the order in which we request Shibboleth attributes to default to less intrusive attributes first, and removes an unused allAttributes field from the database. This requires the following migration steps:

  1. Upgrade to 12.6

  2. Create explicit Shibboleth configuration for existing tenancies by running:

    node etc/migration/12.5-to-12.6/createExplicitShibConfig.js

  3. Use cqlsh to remove the allAttributes by running: cqlsh -k oae -f etc/migration/12.5-to-12.6/allAttributes.cql3

Upgrading from OAE 12.1 to OAE 12.2

OAE 12.2 allows administrators to configure multiple email domains for tenants. A small migration is required to support this:

  1. Use cqlsh to perform the migration that adds the emailDomains column to the Tenant table:

    cqlsh -3 -k oae -f etc/migration/12.1-to-12.2/1.addEmailDomains.cql3

  2. Copy all the email domain values to the new column by executing:

    node etc/migration/12.1-to-12.2/2.tenantEmailDomains.js

  3. Upgrade to 12.2

  4. Drop the emailDomain column (note singular) by executing:

    cqlsh -3 -k oae -f etc/migration/12.1-to-12.2/3.dropEmailDomain.cql3

Upgrading from OAE 11.3 to OAE 12.0

OAE 12 is a major upgrade that introduces a number of significant features:

  • The ability to assign an email domain to a tenant to allow tenants to have "ownership" over an email domain
  • The ability to search users by an exact match of their email address
  • Verification of the authenticity of email address by using a verification email when someone creates / changes their email address
  • The ability to share with people VIA their email address, regardless of whether or not they have an account in the system yet

System Configuration File Upgrades

config.js

Please note the addition of 2 new properties: guestTenantAlias and guestTenantHost. The guestTenantAlias is the tenant alias that will be treated as the "Guest Tenant" for the system. If a tenant with the specified alias does not exist yet, it will be automatically created with the host set in guestTenantHost. After the server starts up and the tenant is created, you can update all guest tenant properties like a normal tenant in the Global Administrative UI after you have started up an app server with the OAE 12 code

nginx.conf

Please note the changes to the nginx.conf between 11.3 and 12.0. In 12.0, the url https://my.oae.site/me no longer exists, it is changed to https://my.oae.site/

For backward compatibility, it is recommended to add a redirect rule to redirect all traffic from /me/* to /*. This is to ensure backward compatibility. Click here to see a sample change made for the Unity production environment.

Infrastructure considerations

Please note that prior to OAE 12.0, it was not necessary to enable the ability for application nodes (Hilary nodes that serve user requests) to send email. Now, with the introduction of email verification, it is important that your application nodes are configured and tested to successfully send mail.

To ensure this is working as expected, log in to your deployed version of OAE 12.0 and set/change an email address in a user profile. A verification email should be sent to the new email address.

Schema Upgrades

To facilitate an upgrade to OAE 12, you should do the following:

  1. Use CQLSH to perform the migration that adds the emailDomain and countryCode property to the Tenant table:

    cqlsh -3 -k oae -f etc/migration/11.3-to-12.0/migration.cql3

  2. Upgrade the Hilary and 3akai-ux code to version 12.0

  3. Shut down Hilary on one of the app servers, and run the node.js script to build the Cassandra email address index used to look up users by email:

    node etc/migration/11.3-to-12.0/mapped-emails.js

    Use node etc/migration/11.3-to-12.0/mapped-emails.js --help to see all usage information

  4. Restart the Hilary service

  5. Rebuild your elasticsearch index data

Global Administrative UI Configurations

The following configurations will be available in the Global Administrative UI after the system has been upgrade. Depending on the service level provided to your tenancies, you may wish to co-ordinate a process to configure tenants immediately after deployment is completed as part of the upgrade process:

Email Domain

Please note the addition of the following new tenant attribute: emailDomain. If you browse to the global administrative UI and inspect the list of tenants, you will see a new "Email Domain" which can be configured for each tenant. This email domain matches as a suffix (e.g., configuring cam.ac.uk matches all of *.cam.ac.uk email address) to determine which tenant a user is redirected to when they are accepting an email invitation to the system.

Country Code

Please note the addition of the following new tenant attribute: countryCode. If you browse to the global administrative UI and inspect the list of tenants, you will see a new "Country" which can be configured for each tenant. This country is only used for aesthetic purposes, maybe if you wanted to display a list of tenants outside the system, you could show a flag!

Guest Tenant

Please note the addition of a "Guest Tenant". This tenant can be used by users who don't have an official affiliation with a tenant owner in the system. You may want to configure the authentication, skin and name of this tenant to match what your instance wants for a public collaboration space. If you do not want users to have an account in this space, please disable authentication to this space.

Upgrading from OAE 11.2 to OAE 11.3

  • There is a new YouTube API Key that must be configured in order to work with the new YouTube APIs. This can be configured in the administrative console Configuration section under OAE Preview Processor > YouTube Configuration

Upgrading from OAE 11.0 to OAE 11.1

  1. Upgrade Hilary and 3akai-ux as usual
  2. Upgrade Etherpad to 9e9207d8b6091375017aba122eb8c3db0fb10e8a
  3. Run the etc/tasks/deleteEtherpadSessions.js script

Upgrading from Elasticsearch 1.1.1 to 1.5.2

  1. Delete the search index (curl -X DELETE http://localhost:9200/oae)
  2. Stop elasticsearch
  3. Uninstall elasticsearch
  4. Install elasticsearch 1.5.2 (we do this through puppet)
  5. Ensure the new elasticsearch service is running
  6. Restart an app server so the search schema gets re-created
  7. Trigger a reindex operation (This can be done from the administration UI > Maintenance > Search)

Upgrading from OAE 10.2 to OAE 10.3

## Deleting groups

It is now possible to delete groups in the system. To support this, you will have to do a couple of things. Before performing the upgrade:

  • drop your search index by executing the following against your ElasticSearch REST layer:

    curl -X DELETE http://localhost:9200/oae

After having performed the upgrade:

  • Run a data migration:

    cqlsh -k <keyspace> -f etc/migration/10.1-to-10.2/migration.cql3

    Note that the migration script is in the incorrect folder. This was an oversight when cutting the release.

  • Re-index your search index. This can be done from the global admin UI in the maintenance panel

Upgrading from OAE 10.1 to OAE 10.2

Google authentication

We're now using Google's recommended Google+ endpoint. This does require you to enable Google+ integration in the Google Developers Console for the API key that you're using.

Upgrading from OAE 10.0 to OAE 10.1

Refresh Memberships Libraries

  • There was a fix that improves how libraries are updated when a user leaves a group. To migrate for this fix, you should open up cqlsh and run the following CQL command:
cqlsh;
> use <keyspace>;
> truncate "LibraryIndex";

Upgrading from OAE 9.2 to OAE 10.0

OAE 10.0 includes some tweaks to the search index. You will have to do the following:

  • Drop your oae search index (e.g., curl -X DELETE http://localhost:9200/oae)
  • Install pdftotext on your PP machines (it's available from poppler-utils)
  • Update both Hilary and 3akai-ux
  • Ensure Hilary's config.js is fully synced up. 10.0 introduces changes to:
    • config.search and
    • config.previews
  • Restart Hilary
  • Re-index your search index. This can be done from the global admin UI in the maintenance panel

Upgrading from OAE 9.1 to OAE 9.2

Data migration

Run the migration script that will truncate the LibraryIndex table. It will be rebuilt on-the-fly:

cqlsh -k <keyspace> -f etc/migration/9.1-to-9.2/truncate-libraries.cql

It's important that you run this script after you've upgraded all your app servers to 9.2.0.

Facebook authentication

Unfortunately, Facebook has made a breaking change to their API which breaks how OAE can identify a user. Users who logged in with Facebook will no longer be able to log in to the same user account. If they log in with Facebook post 9.2, a new account will be created. Combined with the knowledge that the usage of Facebook authentication is very low (almost non-existant) for the known OAE tenancies, no code migration has been written.

Upgrading from OAE 8.0.2 to OAE 9.0

## Data migration

Run the migration script that will add a previews column to the Folders column family on one of your database nodes:

cqlsh -k -f etc/migration/8.0-to-9.0/folders.cql3

Run the migration script that will add a created and createdBy column to the Principals column family from a Hilary folder:

node etc/migration/8.0-to-9.0/groupcreated.js

Tenant Timezone

Timezone configuration has moved from OAE Principals module to OAE Tenants module in the admin interface and needs to be reconfigured for each tenant.

CAS Authentication

The syntax for configuring the mapped attribute values has changed. Rather than using the JSP syntax (<%= attrName %>), attribute names should be specified by enclosing them with curly braces. For example, an old configured value of <%= mail %> should become {mail}.

## Shibboleth

Shibboleth authentication has been completely refactored so that 1 Service Provider entity can support all your tenants. It will require:

  • 1 hostname on which nginx is exposed (for example: shib-sp.example.org)
  • 1 Shibboleth SP entity (for example: https://shib-sp.example.org/shibboleth)
  • Register that SP with your IdP / federations

If you're using Shibboleth authentication you will need to install Apache + mod_shib on your load balancer. See our puppet scripts for information on how to configure the vhost. You will also need to update the configuration settings for the authentication strategy in the application.

Upgrading from OAE 8.0 to OAE 8.0.2

Although not required, installing the Open Sans fonts on the preview processor machines will result in better preview images for collaborative documents.

Upgrading from OAE 7.2 to OAE 8.0

Migration

If you have any "deleted" tenants, you will need to remove them from the database manually. You can list all your tenants by executing:

cqlsh;
> use <keyspace>;
> select * from "Tenant";
     alias | active | displayName | host                     |  deleted  
    -------+--------+-------------+--------------------------+---------
       oae |   True |      Tenant | tenant1.vagrant.oae:8123 |    True
> delete from "Tenant" where "alias"='oae';

You will also need to drop the deleted column from the Tenant columnfamily. This can be done by executing:

cqlsh -k <keyspace> -f etc/migration/7.2-to-8.0/tenants.cql3

CAS Authentication

Tenants using CAS authentication need to change the configuration. The base path has been removed; a full url should be provided. For example: https://login.institution.edu/cas. If your CAS server supports SAML to release user attributes, please tick the useSaml property and set the validationPath to /samlValidate.

Admin Login

Depending on how you configured your default authentication settings, you might need to enable "local" authentication for your admin tenant. This can be done on the command-line with a tool like oaesh.

npm install -g oaesh
oaesh --url https://admin.oae.com --user administrator
config-set -k "oae-authentication/local/enabled=true"

Upgrading from OAE 7.1 to OAE 7.2

Cassandra / DataStax Enterprise upgrade

Although not technically required, the OAE team now recommends that you upgrade your Cassandra installation to 2.0.x (tested with 2.0.8) or if you're running DSE upgrade to 4.5.x (tested with 4.5.0)

Etherpad Upgrade

You will need to upgrade your etherpad instances to an unreleased version. This will be 1.4.0 + an extra set of commits.

You will also need to upgrade the ep_oae etherpad plugin to 8.0.0. Depending on where you run your RabbitMQ servers, you'll need to tweak Etherpad's settings.json file. See the Etherpad lite section in Hilary's README for more information.

Upgrading from OAE 6.5 to OAE 7.0

CQL Migration

OAE 7.0 adds a new property to user profiles: emailPreference. It needs to be added to the user profile by running the following schema change using cqlsh before you start up Hilary version 7.0:

ALTER TABLE "Principals" ADD "emailPreference" text;

Upgrading from OAE 6.4 to OAE 6.5

OAE 6.5 brings compatibility with ElasticSearch 1.1 which is non-backwards compatible with ElasticSearch 0.20.7. Upgrading can be done by performing these steps:

  1. Delete the search index on all search nodes by removing it on disk (rm -rf /data/elasticsearch)

  2. Uninstall elasticsearch 0.20.7 on all nodes (dpkg -r elasticsearch)

  3. Bring down the entire search cluster (service elasticsearch stop)

  4. Install elasticsearch 1.1 on all nodes through the new apt repository

  5. Bring the cluster back up

  6. Install Hilary 6.5 on a single app node and restart the Hilary service. This will create the new elasticsearch index and mapping

  7. Install Hilary 6.5 on your other Hilary machines and restart the Hilary service.

  8. Trigger a full re-index by logging into the global admin console and executing the following in the Javascript console:

    $.ajax({'type': 'POST', 'url': '/api/search/reindexAll'})

An issue has been identified where not all content is fully re-indexed. In case this happens in your environment, it can usually be remedied by running the above reindex command a few times.

Upgrading from OAE 5.0 to OAE 6.0

Responsive UI

OAE 6.0 brings a new responsive UI that introduces new skinning values. We recommend that these are configured for each tenant in the Skinning section of the Admin UI:

  • Small institutional logo: The institution logo to be used on mobile devices
  • Authentication buttons: Alter the appearance of the Shibboleth and CAS authentication buttons

Config.js

The name of the config.previews.dir property in Hilary's config.js file has been changed to config.previews.tmpDir. When working from a custom version of the config.js file, this should be updated.

Activity streams

OAE 4 introduced Push Notifications which changed the way activity streams were stored in Cassandra. Some migration code was included that ensured that older activity streams could still be read. This code has been removed in OAE 6.0. As the config.activity.activityTtl property in Hilary's config.js file determines the amount of time activities are persisted in Cassandra, you will have to run 4.x or 5.0 for at least this amount of time in production. After that amount of time it is safe to deploy 6.0. If you upgrade directly from 3.X to a version after 5.0, your activity streams will become empty in the user interface.

Upgrading from OAE 4.4 to OAE 5.0

OAE 5.0 will introduce an upgrade from CQL2 to CQL3. You should first upgrade to OAE 4.4 before upgrading to OAE 5.0.

The upgrade procedure for 5.0 is outlined below.

CQL3 Production Upgrade Procedure

To upgrade to OAE 5.0 from 4.4, a set of scripts are provided in the directory:

etc/migration/cql2_to_cql3/1-upgrade-cql2-to-cql3

To upgrade the Cassandra schema to CQL3, perform the following steps:

  1. Use cassandra-cli to apply an update to the Revision table metadata:

    cassandra-cli -k oae -f etc/migration/cql2_to_cql3/1-upgrade-cql2-to-cql3/1-cassandra-cli

  2. Use CQLSH to perform CQL3 updates to prepare it for running CQL3 Hilary queries:

    cqlsh -3 -k oae -f etc/migration/cql2_to_cql3/1-upgrade-cql2-to-cql3/2-cqlsh.cql3

Once the schema upgrade is complete, you can start up your OAE 5.0 Hilary nodes and you are finished.

Upgrade Testing Procedure

To help test the upgrade in a local development environment, the following tools are provided:

  1. There is a CQL2 schema file etc/migration/cql2_to_cql3/1-upgrade-cql2-to-cql3/create_cql2_prod_schema.cql2 available that represents a schema dump of our production CQL2 schema before migration. To be detailed, you can replace this schema file with the output of the command:

    $ cqlsh -2
Connected to test at localhost:9160.
[cqlsh 3.1.7 | Cassandra 0.0.0 | CQL spec 2.0.0 | Thrift protocol 19.36.1]
Use HELP for help.
cqlsh> describe keyspace oae;

CREATE KEYSPACE oae WITH strategy_class = 'SimpleStrategy'
  AND strategy_options:replication_factor = '1';

USE oae;

CREATE TABLE Config (
  tenantAlias text PRIMARY KEY
) WITH
  comment='' AND
  comparator=text AND
  read_repair_chance=0.100000 AND
  gc_grace_seconds=864000 AND
  default_validation=text AND
  min_compaction_threshold=4 AND
  max_compaction_threshold=32 AND
  replicate_on_write='true' AND
  compaction_strategy_class='SizeTieredCompactionStrategy' AND
  populate_io_cache_on_flush='false' AND
  compression_parameters:sstable_compression='SnappyCompressor';

CREATE TABLE OAuthAccessToken (
  token text PRIMARY KEY,
  userId text,
  clientId text
) WITH
  comment='' AND
  comparator=text AND
  read_repair_chance=0.100000 AND
  gc_grace_seconds=864000 AND
  default_validation=text AND
  min_compaction_threshold=4 AND

...

    Then replace only the CREATE TABLE commands of the create_cql2_prod_schema.cql2 file with the ones from the command output.

  2. Once you have placed a copy of your production schema into the file, you can run the following command from your local Hilary directory: etc/migration/cql2_to_cql3/unit_test_migration.sh etc/migration/cql2_to_cql3/create_cql2_prod_schema.cql2 to invoke the unit tests on an upgraded version of the CQL3 schema. Note that unit tests should not be run on a production database. Only run it in a local development environment. The script will first apply the CQL2 version of your schema, then perform the upgrade procedure on it, then start the Hilary server and run a full battery of tests to ensure that all application functionality is correct.

  3. Test in a staging environment. While the unit tests ensure that all application functionality still works on an upgraded copy of your schema, you should also restore a mirror of your application into a secure test environment and perform a smoke test on your content to ensure application data upgrades successfully.

Upgrading from OAE 4.3 to OAE 4.4

OAE 4.4 introduces search indexing and preview processing enhancements that require immediate attention when upgrading.

  1. You will need to rebuild your search index as part of your upgrade to apply a new search schema

  2. You will need to [reprocess previews] of all revisions of Word Document / PDF Document content items using the following command in the Admin UI browser developer console:

    $.ajax({
    'type': 'POST',
    'url': '/api/content/reprocessPreviews',
    'data': {
        'revision_mime': [
            'application/msword',
            'application/rdf+xml',
            'application/vnd.ms-excel',
            'application/vnd.ms-powerpoint',
            'application/vnd.oasis.opendocument.presentation',
            'application/vnd.oasis.opendocument.spreadsheet',
            'application/vnd.oasis.opendocument.text',
            'application/vnd.openxmlformats-officedocument.presentationml.presentation',
            'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
            'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
            'application/x-mspowerpoint',
            'application/x-pdf',
            'application/x-powerpoint',
            'text/plain',
            'application/pdf'
        ]
    }
});

Next Release

This is a placeholder for un-released work that has actually been merged to master. All these items are committed to have to be performed for the next OAE release.

Work In Progress

This is a placeholder for un-released works in progress that will require migration when they are finally released. None of these steps need to be completed until they appear under an actual upgrade section.

Add a custom footer
Add a custom sidebar
Clone this wiki locally