-
Notifications
You must be signed in to change notification settings - Fork 96
OAE Upgrade Guide
- Upgrading from OAE 12.1 to 12.2
- Upgrading from OAE 11.3 to 12.0
- Upgrading from OAE 11.2 to 11.3
- Upgrading from OAE 11.0 to 11.1
- Upgrading Elasticsearch from 1.1.1 to 1.5.2
- Upgrading from OAE 10.2 to 10.3
- Upgrading from OAE 10.1 to 10.2
- Upgrading from OAE 10.0 to 10.1
- Upgrading from OAE 9.2 to 10.0
- Upgrading from OAE 9.1 to 9.2
- Upgrading from OAE 8.0 to 9.0
- Upgrading from OAE 8.0 to 8.0.2
- Upgrading from OAE 7.2 to OAE 8.0
- Upgrading from OAE 7.1 to OAE 7.2
- Upgrading from OAE 6.5 to OAE 7.0
- Upgrading from OAE 6.4 to OAE 6.5
- Upgrading from OAE 5.0 to OAE 6.0
- Upgrading from OAE 4.4 to OAE 5.0
- Upgrading from OAE 4.3 to OAE 4.4
OAE 12.2 allows administrators to configure multiple email domains for tenants. A small migration is required to support this:
-
Use cqlsh to perform the migration that adds the
emailDomainscolumn to theTenanttable:cqlsh -3 -k oae -f etc/migration/12.1-to-12.2/1.addEmailDomains.cql3 -
Copy all the email domain values to the new column by executing:
node etc/migration/12.1-to-12.2/2.tenantEmailDomains.js -
Upgrade to 12.2
-
Drop the
emailDomaincolumn (note singular) by executing:cqlsh -3 -k oae -f etc/migration/12.1-to-12.2/3.dropEmailDomain.cql3
OAE 12 is a major upgrade that introduces a number of significant features:
- The ability to assign an email domain to a tenant to allow tenants to have "ownership" over an email domain
- The ability to search users by an exact match of their email address
- Verification of the authenticity of email address by using a verification email when someone creates / changes their email address
- The ability to share with people VIA their email address, regardless of whether or not they have an account in the system yet
config.js
Please note the addition of 2 new properties: guestTenantAlias and guestTenantHost. The guestTenantAlias is the tenant alias that will be treated as the "Guest Tenant" for the system. If a tenant with the specified alias does not exist yet, it will be automatically created with the host set in guestTenantHost. After the server starts up and the tenant is created, you can update all guest tenant properties like a normal tenant in the Global Administrative UI after you have started up an app server with the OAE 12 code
nginx.conf
Please note the changes to the nginx.conf between 11.3 and 12.0. In 12.0, the url https://my.oae.site/me no longer exists, it is changed to https://my.oae.site/
For backward compatibility, it is recommended to add a redirect rule to redirect all traffic from /me/* to /*. This is to ensure backward compatibility. Click here to see a sample change made for the Unity production environment.
Infrastructure considerations
Please note that prior to OAE 12.0, it was not necessary to enable the ability for application nodes (Hilary nodes that serve user requests) to send email. Now, with the introduction of email verification, it is important that your application nodes are configured and tested to successfully send mail.
To ensure this is working as expected, log in to your deployed version of OAE 12.0 and set/change an email address in a user profile. A verification email should be sent to the new email address.
To facilitate an upgrade to OAE 12, you should do the following:
-
Use CQLSH to perform the migration that adds the
emailDomainandcountryCodeproperty to theTenanttable:cqlsh -3 -k oae -f etc/migration/11.3-to-12.0/migration.cql3 -
Upgrade the Hilary and 3akai-ux code to version 12.0
-
Shut down Hilary on one of the app servers, and run the node.js script to build the Cassandra email address index used to look up users by email:
node etc/migration/11.3-to-12.0/mapped-emails.jsUse
node etc/migration/11.3-to-12.0/mapped-emails.js --helpto see all usage information -
Restart the Hilary service
The following configurations will be available in the Global Administrative UI after the system has been upgrade. Depending on the service level provided to your tenancies, you may wish to co-ordinate a process to configure tenants immediately after deployment is completed as part of the upgrade process:
Email Domain
Please note the addition of the following new tenant attribute: emailDomain. If you browse to the global administrative UI and inspect the list of tenants, you will see a new "Email Domain" which can be configured for each tenant. This email domain matches as a suffix (e.g., configuring cam.ac.uk matches all of *.cam.ac.uk email address) to determine which tenant a user is redirected to when they are accepting an email invitation to the system.
Country Code
Please note the addition of the following new tenant attribute: countryCode. If you browse to the global administrative UI and inspect the list of tenants, you will see a new "Country" which can be configured for each tenant. This country is only used for aesthetic purposes, maybe if you wanted to display a list of tenants outside the system, you could show a flag!
Guest Tenant
Please note the addition of a "Guest Tenant". This tenant can be used by users who don't have an official affiliation with a tenant owner in the system. You may want to configure the authentication, skin and name of this tenant to match what your instance wants for a public collaboration space. If you do not want users to have an account in this space, please disable authentication to this space.
- There is a new YouTube API Key that must be configured in order to work with the new YouTube APIs. This can be configured in the administrative console
Configurationsection underOAE Preview Processor > YouTube Configuration
- Upgrade Hilary and 3akai-ux as usual
- Upgrade Etherpad to 9e9207d8b6091375017aba122eb8c3db0fb10e8a
- Run the
etc/tasks/deleteEtherpadSessions.jsscript
- Delete the search index (curl -X DELETE http://localhost:9200/oae)
- Stop elasticsearch
- Uninstall elasticsearch
- Install elasticsearch 1.5.2 (we do this through puppet)
- Ensure the new elasticsearch service is running
- Restart an app server so the search schema gets re-created
- Trigger a reindex operation (This can be done from the administration UI > Maintenance > Search)
## Deleting groups
It is now possible to delete groups in the system. To support this, you will have to do a couple of things. Before performing the upgrade:
-
drop your search index by executing the following against your ElasticSearch REST layer:
curl -X DELETE http://localhost:9200/oae
After having performed the upgrade:
-
Run a data migration:
cqlsh -k <keyspace> -f etc/migration/10.1-to-10.2/migration.cql3Note that the migration script is in the incorrect folder. This was an oversight when cutting the release.
-
Re-index your search index. This can be done from the global admin UI in the maintenance panel
We're now using Google's recommended Google+ endpoint. This does require you to enable Google+ integration in the Google Developers Console for the API key that you're using.
- There was a fix that improves how libraries are updated when a user leaves a group. To migrate for this fix, you should open up
cqlshand run the following CQL command:
cqlsh;
> use <keyspace>;
> truncate "LibraryIndex";
OAE 10.0 includes some tweaks to the search index. You will have to do the following:
- Drop your oae search index (e.g.,
curl -X DELETE http://localhost:9200/oae) - Install pdftotext on your PP machines (it's available from poppler-utils)
- Update both Hilary and 3akai-ux
- Ensure Hilary's config.js is fully synced up. 10.0 introduces changes to:
- config.search and
- config.previews
- Restart Hilary
- Re-index your search index. This can be done from the global admin UI in the maintenance panel
Run the migration script that will truncate the LibraryIndex table. It will be rebuilt on-the-fly:
cqlsh -k <keyspace> -f etc/migration/9.1-to-9.2/truncate-libraries.cql
It's important that you run this script after you've upgraded all your app servers to 9.2.0.
Unfortunately, Facebook has made a breaking change to their API which breaks how OAE can identify a user. Users who logged in with Facebook will no longer be able to log in to the same user account. If they log in with Facebook post 9.2, a new account will be created. Combined with the knowledge that the usage of Facebook authentication is very low (almost non-existant) for the known OAE tenancies, no code migration has been written.
## Data migration
Run the migration script that will add a previews column to the Folders column family on one of your database nodes:
cqlsh -k -f etc/migration/8.0-to-9.0/folders.cql3
Run the migration script that will add a created and createdBy column to the Principals column family from a Hilary folder:
node etc/migration/8.0-to-9.0/groupcreated.js
Timezone configuration has moved from OAE Principals module to OAE Tenants module in the admin interface and needs to be reconfigured for each tenant.
The syntax for configuring the mapped attribute values has changed. Rather than using the JSP syntax (<%= attrName %>), attribute names should be specified by enclosing them with curly braces.
For example, an old configured value of <%= mail %> should become {mail}.
## Shibboleth
Shibboleth authentication has been completely refactored so that 1 Service Provider entity can support all your tenants. It will require:
- 1 hostname on which nginx is exposed (for example: shib-sp.example.org)
- 1 Shibboleth SP entity (for example: https://shib-sp.example.org/shibboleth)
- Register that SP with your IdP / federations
If you're using Shibboleth authentication you will need to install Apache + mod_shib on your load balancer. See our puppet scripts for information on how to configure the vhost. You will also need to update the configuration settings for the authentication strategy in the application.
Although not required, installing the Open Sans fonts on the preview processor machines will result in better preview images for collaborative documents.
If you have any "deleted" tenants, you will need to remove them from the database manually. You can list all your tenants by executing:
cqlsh;
> use <keyspace>;
> select * from "Tenant";
alias | active | displayName | host | deleted
-------+--------+-------------+--------------------------+---------
oae | True | Tenant | tenant1.vagrant.oae:8123 | True
> delete from "Tenant" where "alias"='oae';
You will also need to drop the deleted column from the Tenant columnfamily. This can be done by executing:
cqlsh -k <keyspace> -f etc/migration/7.2-to-8.0/tenants.cql3
Tenants using CAS authentication need to change the configuration. The base path has been removed; a full url should be provided. For example: https://login.institution.edu/cas. If your CAS server supports SAML to release user attributes, please tick the useSaml property and set the validationPath to /samlValidate.
Depending on how you configured your default authentication settings, you might need to enable "local" authentication for your admin tenant. This can be done on the command-line with a tool like oaesh.
npm install -g oaesh
oaesh --url https://admin.oae.com --user administrator
config-set -k "oae-authentication/local/enabled=true"
Although not technically required, the OAE team now recommends that you upgrade your Cassandra installation to 2.0.x (tested with 2.0.8) or if you're running DSE upgrade to 4.5.x (tested with 4.5.0)
You will need to upgrade your etherpad instances to an unreleased version. This will be 1.4.0 + an extra set of commits.
You will also need to upgrade the ep_oae etherpad plugin to 8.0.0. Depending on where you run your RabbitMQ servers, you'll need to tweak Etherpad's settings.json file. See the Etherpad lite section in Hilary's README for more information.
OAE 7.0 adds a new property to user profiles: emailPreference. It needs to be added to the user profile by running the following schema change using cqlsh before you start up Hilary version 7.0:
ALTER TABLE "Principals" ADD "emailPreference" text;OAE 6.5 brings compatibility with ElasticSearch 1.1 which is non-backwards compatible with ElasticSearch 0.20.7. Upgrading can be done by performing these steps:
-
Delete the search index on all search nodes by removing it on disk (
rm -rf /data/elasticsearch) -
Uninstall elasticsearch 0.20.7 on all nodes (
dpkg -r elasticsearch) -
Bring down the entire search cluster (
service elasticsearch stop) -
Install elasticsearch 1.1 on all nodes through the new apt repository
-
Bring the cluster back up
-
Install Hilary 6.5 on a single app node and restart the Hilary service. This will create the new elasticsearch index and mapping
-
Install Hilary 6.5 on your other Hilary machines and restart the Hilary service.
-
Trigger a full re-index by logging into the global admin console and executing the following in the Javascript console:
$.ajax({'type': 'POST', 'url': '/api/search/reindexAll'})
An issue has been identified where not all content is fully re-indexed. In case this happens in your environment, it can usually be remedied by running the above reindex command a few times.
OAE 6.0 brings a new responsive UI that introduces new skinning values. We recommend that these are configured for each tenant in the Skinning section of the Admin UI:
-
Small institutional logo: The institution logo to be used on mobile devices -
Authentication buttons: Alter the appearance of the Shibboleth and CAS authentication buttons
The name of the config.previews.dir property in Hilary's config.js file has been changed to config.previews.tmpDir. When working from a custom version of the config.js file, this should be updated.
OAE 4 introduced Push Notifications which changed the way activity streams were stored in Cassandra. Some migration code was included that ensured that older activity streams could still be read. This code has been removed in OAE 6.0. As the config.activity.activityTtl property in Hilary's config.js file determines the amount of time activities are persisted in Cassandra, you will have to run 4.x or 5.0 for at least this amount of time in production. After that amount of time it is safe to deploy 6.0. If you upgrade directly from 3.X to a version after 5.0, your activity streams will become empty in the user interface.
OAE 5.0 will introduce an upgrade from CQL2 to CQL3. You should first upgrade to OAE 4.4 before upgrading to OAE 5.0.
The upgrade procedure for 5.0 is outlined below.
To upgrade to OAE 5.0 from 4.4, a set of scripts are provided in the directory:
etc/migration/cql2_to_cql3/1-upgrade-cql2-to-cql3
To upgrade the Cassandra schema to CQL3, perform the following steps:
-
Use
cassandra-clito apply an update to theRevisiontable metadata:cassandra-cli -k oae -f etc/migration/cql2_to_cql3/1-upgrade-cql2-to-cql3/1-cassandra-cli -
Use CQLSH to perform CQL3 updates to prepare it for running CQL3 Hilary queries:
cqlsh -3 -k oae -f etc/migration/cql2_to_cql3/1-upgrade-cql2-to-cql3/2-cqlsh.cql3
Once the schema upgrade is complete, you can start up your OAE 5.0 Hilary nodes and you are finished.
To help test the upgrade in a local development environment, the following tools are provided:
-
There is a CQL2 schema file
etc/migration/cql2_to_cql3/1-upgrade-cql2-to-cql3/create_cql2_prod_schema.cql2available that represents a schema dump of our production CQL2 schema before migration. To be detailed, you can replace this schema file with the output of the command:$ cqlsh -2 Connected to test at localhost:9160. [cqlsh 3.1.7 | Cassandra 0.0.0 | CQL spec 2.0.0 | Thrift protocol 19.36.1] Use HELP for help. cqlsh> describe keyspace oae; CREATE KEYSPACE oae WITH strategy_class = 'SimpleStrategy' AND strategy_options:replication_factor = '1'; USE oae; CREATE TABLE Config ( tenantAlias text PRIMARY KEY ) WITH comment='' AND comparator=text AND read_repair_chance=0.100000 AND gc_grace_seconds=864000 AND default_validation=text AND min_compaction_threshold=4 AND max_compaction_threshold=32 AND replicate_on_write='true' AND compaction_strategy_class='SizeTieredCompactionStrategy' AND populate_io_cache_on_flush='false' AND compression_parameters:sstable_compression='SnappyCompressor'; CREATE TABLE OAuthAccessToken ( token text PRIMARY KEY, userId text, clientId text ) WITH comment='' AND comparator=text AND read_repair_chance=0.100000 AND gc_grace_seconds=864000 AND default_validation=text AND min_compaction_threshold=4 AND ...Then replace only the CREATE TABLE commands of the
create_cql2_prod_schema.cql2file with the ones from the command output. -
Once you have placed a copy of your production schema into the file, you can run the following command from your local Hilary directory:
etc/migration/cql2_to_cql3/unit_test_migration.sh etc/migration/cql2_to_cql3/create_cql2_prod_schema.cql2to invoke the unit tests on an upgraded version of the CQL3 schema. Note that unit tests should not be run on a production database. Only run it in a local development environment. The script will first apply the CQL2 version of your schema, then perform the upgrade procedure on it, then start the Hilary server and run a full battery of tests to ensure that all application functionality is correct. -
Test in a staging environment. While the unit tests ensure that all application functionality still works on an upgraded copy of your schema, you should also restore a mirror of your application into a secure test environment and perform a smoke test on your content to ensure application data upgrades successfully.
OAE 4.4 introduces search indexing and preview processing enhancements that require immediate attention when upgrading.
-
You will need to rebuild your search index as part of your upgrade to apply a new search schema
-
You will need to [reprocess previews] of all revisions of Word Document / PDF Document content items using the following command in the Admin UI browser developer console:
$.ajax({ 'type': 'POST', 'url': '/api/content/reprocessPreviews', 'data': { 'revision_mime': [ 'application/msword', 'application/rdf+xml', 'application/vnd.ms-excel', 'application/vnd.ms-powerpoint', 'application/vnd.oasis.opendocument.presentation', 'application/vnd.oasis.opendocument.spreadsheet', 'application/vnd.oasis.opendocument.text', 'application/vnd.openxmlformats-officedocument.presentationml.presentation', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', 'application/x-mspowerpoint', 'application/x-pdf', 'application/x-powerpoint', 'text/plain', 'application/pdf' ] } });
This is a placeholder for un-released work that has actually been merged to master. All these items are committed to have to be performed for the next OAE release.
This is a placeholder for un-released works in progress that will require migration when they are finally released. None of these steps need to be completed until they appear under an actual upgrade section.