msan694_disco/examples/Week2/ex1.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "lines = sc.textFile(\"../Data/ignatian_pedagogy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'= Ignatian Values =',\n",
       " u'The University of San Francisco enjoys a distinguished heritage and Jesuit tradition.  At the core of this tradition are transcendent values, including the integration of learning, faith and service; care for the whole person; character and conviction; religious truth and interfaith understanding; and a commitment to building a more just world.  The key values of this Jesuit tradition are as follows:',\n",
       " u'***********************************************************************************',\n",
       " u\"1. Contemplative in Action - St. Ignatius Loyola believed that prayer and reflectivity should so guide our choices and actions that our activity itself becomes a way of entering into union with and praising God.  Being a contemplative in action also means seeing beyond the superficial in life to appreciate the mystery, beauty, and sacredness of all life.  It is a means of seeing God in all things and in everyone.  Contemplation is a critical dimension of the spiritual life and it is reflected in USF's commitment to prayer and spiritual growth.  Analogously, in the academic life, a spirit of reflectivity is a critical aspect of intellectual inquiry.\",\n",
       " u'',\n",
       " u'2. Academic Excellence - In 1547, the first Jesuits were invited to begin a college in Messina, Italy, so that the young men of that town could receive the same quality of education that the early Jesuits promoted in training their own men.  USF is a descendant of this original Jesuit commitment to education.  Academic excellence describes the great importance that Jesuits have placed on the life of the mind as a means for uncovering truth and discovering meaning.  It is this tradition that imbues the other 27 Jesuit universities and colleges, as well as the Jesuit high schools and middle schools in the US.',\n",
       " u'',\n",
       " u\"3. Educating the Whole Person - St. Ignatius believed that God could be discovered in every human endeavor, in every facet of learning and experience, and in every field of study.  Consequently, he promoted the development of the spiritual, intellectual, artistic, social, and physical aspects of each person.  The commitment to educating the whole person is evident in USF's strong core curriculum and its wide array of learning activities and experiences that bridges cultures and national boundaries.\",\n",
       " u'',\n",
       " u'4. \"Cura Personalis\" - \"Care of the Person\" was originally used to describe the responsibility of the Jesuit Superior to care for each man in the community with his unique gifts, challenges, needs, and possibilities.  This value now is applied more broadly to include the relationship between educators and students, as well as the professional relationships among all those who work in the University.  \"Cura Personalis\" suggests individualized attention to the needs of the other, distinct respect for his or her unique circumstances and concerns, and an appropriate appreciation for his or her particular gifts and insights.',\n",
       " u'',\n",
       " u'5. Women and Men for Others - Fr. Pedro Arrupe, S.J., Superior General of the Society of Jesuis from 1965 to 1981, employed the phrase \"Men for Others\" in a notable 1973 presentation in Valencia, Spain.  Father Arrupe provocatively challenged the alumni of Jesuit schools and universities to be engaged in the struggle for justice to protect the needs of those people who are most vulnerable.  Today this phrase has become more inclusive and its spirit is evidenced in USF\\'s promotion of service projects; justice learning through immersion; and its challenge to students, faculty, staff, and alumni to \"Change the World from Here.\"',\n",
       " u'',\n",
       " u'6. Interreligious Understanding - Reflecting themes from the Second Vatican Council, the 34th General Congregation of the Society of Jesus made a significant commitment to ecumenical and interreligious engagement and understanding.  The USF community comprises a wide variety of religious traditions; and through its University Ministry, it encourages people of different faiths-and no faith-to create an inclusive community that is reflective and authentic in its search for meaning and truth.',\n",
       " u'',\n",
       " u\"7. Community in Diversity - As a Catholic and Jesuit University, USF welcomes and sustains the rich diversity among the students, faculty, and staff that comprises its community.  The USF community continues to be enriched by people of different abilities, ages, colors, creeds, faiths, philosophies, gender orientations, cultures, nationalities, family backgrounds, and educational experiences.  U.S. News and World Report ranks USF among the nation's top 30 university for student ethnic diversity and international student enrollment.\",\n",
       " u'***********************************************************************************']"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lines.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example 1-1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "words = lines.map(lambda line : line.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[u'=', u'Ignatian', u'Values', u'='],\n",
       " [u'The',\n",
       "  u'University',\n",
       "  u'of',\n",
       "  u'San',\n",
       "  u'Francisco',\n",
       "  u'enjoys',\n",
       "  u'a',\n",
       "  u'distinguished',\n",
       "  u'heritage',\n",
       "  u'and',\n",
       "  u'Jesuit',\n",
       "  u'tradition.',\n",
       "  u'At',\n",
       "  u'the',\n",
       "  u'core',\n",
       "  u'of',\n",
       "  u'this',\n",
       "  u'tradition',\n",
       "  u'are',\n",
       "  u'transcendent',\n",
       "  u'values,',\n",
       "  u'including',\n",
       "  u'the',\n",
       "  u'integration',\n",
       "  u'of',\n",
       "  u'learning,',\n",
       "  u'faith',\n",
       "  u'and',\n",
       "  u'service;',\n",
       "  u'care',\n",
       "  u'for',\n",
       "  u'the',\n",
       "  u'whole',\n",
       "  u'person;',\n",
       "  u'character',\n",
       "  u'and',\n",
       "  u'conviction;',\n",
       "  u'religious',\n",
       "  u'truth',\n",
       "  u'and',\n",
       "  u'interfaith',\n",
       "  u'understanding;',\n",
       "  u'and',\n",
       "  u'a',\n",
       "  u'commitment',\n",
       "  u'to',\n",
       "  u'building',\n",
       "  u'a',\n",
       "  u'more',\n",
       "  u'just',\n",
       "  u'world.',\n",
       "  u'The',\n",
       "  u'key',\n",
       "  u'values',\n",
       "  u'of',\n",
       "  u'this',\n",
       "  u'Jesuit',\n",
       "  u'tradition',\n",
       "  u'are',\n",
       "  u'as',\n",
       "  u'follows:'],\n",
       " [u'***********************************************************************************'],\n",
       " [u'1.',\n",
       "  u'Contemplative',\n",
       "  u'in',\n",
       "  u'Action',\n",
       "  u'-',\n",
       "  u'St.',\n",
       "  u'Ignatius',\n",
       "  u'Loyola',\n",
       "  u'believed',\n",
       "  u'that',\n",
       "  u'prayer',\n",
       "  u'and',\n",
       "  u'reflectivity',\n",
       "  u'should',\n",
       "  u'so',\n",
       "  u'guide',\n",
       "  u'our',\n",
       "  u'choices',\n",
       "  u'and',\n",
       "  u'actions',\n",
       "  u'that',\n",
       "  u'our',\n",
       "  u'activity',\n",
       "  u'itself',\n",
       "  u'becomes',\n",
       "  u'a',\n",
       "  u'way',\n",
       "  u'of',\n",
       "  u'entering',\n",
       "  u'into',\n",
       "  u'union',\n",
       "  u'with',\n",
       "  u'and',\n",
       "  u'praising',\n",
       "  u'God.',\n",
       "  u'Being',\n",
       "  u'a',\n",
       "  u'contemplative',\n",
       "  u'in',\n",
       "  u'action',\n",
       "  u'also',\n",
       "  u'means',\n",
       "  u'seeing',\n",
       "  u'beyond',\n",
       "  u'the',\n",
       "  u'superficial',\n",
       "  u'in',\n",
       "  u'life',\n",
       "  u'to',\n",
       "  u'appreciate',\n",
       "  u'the',\n",
       "  u'mystery,',\n",
       "  u'beauty,',\n",
       "  u'and',\n",
       "  u'sacredness',\n",
       "  u'of',\n",
       "  u'all',\n",
       "  u'life.',\n",
       "  u'It',\n",
       "  u'is',\n",
       "  u'a',\n",
       "  u'means',\n",
       "  u'of',\n",
       "  u'seeing',\n",
       "  u'God',\n",
       "  u'in',\n",
       "  u'all',\n",
       "  u'things',\n",
       "  u'and',\n",
       "  u'in',\n",
       "  u'everyone.',\n",
       "  u'Contemplation',\n",
       "  u'is',\n",
       "  u'a',\n",
       "  u'critical',\n",
       "  u'dimension',\n",
       "  u'of',\n",
       "  u'the',\n",
       "  u'spiritual',\n",
       "  u'life',\n",
       "  u'and',\n",
       "  u'it',\n",
       "  u'is',\n",
       "  u'reflected',\n",
       "  u'in',\n",
       "  u\"USF's\",\n",
       "  u'commitment',\n",
       "  u'to',\n",
       "  u'prayer',\n",
       "  u'and',\n",
       "  u'spiritual',\n",
       "  u'growth.',\n",
       "  u'Analogously,',\n",
       "  u'in',\n",
       "  u'the',\n",
       "  u'academic',\n",
       "  u'life,',\n",
       "  u'a',\n",
       "  u'spirit',\n",
       "  u'of',\n",
       "  u'reflectivity',\n",
       "  u'is',\n",
       "  u'a',\n",
       "  u'critical',\n",
       "  u'aspect',\n",
       "  u'of',\n",
       "  u'intellectual',\n",
       "  u'inquiry.'],\n",
       " [],\n",
       " [u'2.',\n",
       "  u'Academic',\n",
       "  u'Excellence',\n",
       "  u'-',\n",
       "  u'In',\n",
       "  u'1547,',\n",
       "  u'the',\n",
       "  u'first',\n",
       "  u'Jesuits',\n",
       "  u'were',\n",
       "  u'invited',\n",
       "  u'to',\n",
       "  u'begin',\n",
       "  u'a',\n",
       "  u'college',\n",
       "  u'in',\n",
       "  u'Messina,',\n",
       "  u'Italy,',\n",
       "  u'so',\n",
       "  u'that',\n",
       "  u'the',\n",
       "  u'young',\n",
       "  u'men',\n",
       "  u'of',\n",
       "  u'that',\n",
       "  u'town',\n",
       "  u'could',\n",
       "  u'receive',\n",
       "  u'the',\n",
       "  u'same',\n",
       "  u'quality',\n",
       "  u'of',\n",
       "  u'education',\n",
       "  u'that',\n",
       "  u'the',\n",
       "  u'early',\n",
       "  u'Jesuits',\n",
       "  u'promoted',\n",
       "  u'in',\n",
       "  u'training',\n",
       "  u'their',\n",
       "  u'own',\n",
       "  u'men.',\n",
       "  u'USF',\n",
       "  u'is',\n",
       "  u'a',\n",
       "  u'descendant',\n",
       "  u'of',\n",
       "  u'this',\n",
       "  u'original',\n",
       "  u'Jesuit',\n",
       "  u'commitment',\n",
       "  u'to',\n",
       "  u'education.',\n",
       "  u'Academic',\n",
       "  u'excellence',\n",
       "  u'describes',\n",
       "  u'the',\n",
       "  u'great',\n",
       "  u'importance',\n",
       "  u'that',\n",
       "  u'Jesuits',\n",
       "  u'have',\n",
       "  u'placed',\n",
       "  u'on',\n",
       "  u'the',\n",
       "  u'life',\n",
       "  u'of',\n",
       "  u'the',\n",
       "  u'mind',\n",
       "  u'as',\n",
       "  u'a',\n",
       "  u'means',\n",
       "  u'for',\n",
       "  u'uncovering',\n",
       "  u'truth',\n",
       "  u'and',\n",
       "  u'discovering',\n",
       "  u'meaning.',\n",
       "  u'It',\n",
       "  u'is',\n",
       "  u'this',\n",
       "  u'tradition',\n",
       "  u'that',\n",
       "  u'imbues',\n",
       "  u'the',\n",
       "  u'other',\n",
       "  u'27',\n",
       "  u'Jesuit',\n",
       "  u'universities',\n",
       "  u'and',\n",
       "  u'colleges,',\n",
       "  u'as',\n",
       "  u'well',\n",
       "  u'as',\n",
       "  u'the',\n",
       "  u'Jesuit',\n",
       "  u'high',\n",
       "  u'schools',\n",
       "  u'and',\n",
       "  u'middle',\n",
       "  u'schools',\n",
       "  u'in',\n",
       "  u'the',\n",
       "  u'US.'],\n",
       " [],\n",
       " [u'3.',\n",
       "  u'Educating',\n",
       "  u'the',\n",
       "  u'Whole',\n",
       "  u'Person',\n",
       "  u'-',\n",
       "  u'St.',\n",
       "  u'Ignatius',\n",
       "  u'believed',\n",
       "  u'that',\n",
       "  u'God',\n",
       "  u'could',\n",
       "  u'be',\n",
       "  u'discovered',\n",
       "  u'in',\n",
       "  u'every',\n",
       "  u'human',\n",
       "  u'endeavor,',\n",
       "  u'in',\n",
       "  u'every',\n",
       "  u'facet',\n",
       "  u'of',\n",
       "  u'learning',\n",
       "  u'and',\n",
       "  u'experience,',\n",
       "  u'and',\n",
       "  u'in',\n",
       "  u'every',\n",
       "  u'field',\n",
       "  u'of',\n",
       "  u'study.',\n",
       "  u'Consequently,',\n",
       "  u'he',\n",
       "  u'promoted',\n",
       "  u'the',\n",
       "  u'development',\n",
       "  u'of',\n",
       "  u'the',\n",
       "  u'spiritual,',\n",
       "  u'intellectual,',\n",
       "  u'artistic,',\n",
       "  u'social,',\n",
       "  u'and',\n",
       "  u'physical',\n",
       "  u'aspects',\n",
       "  u'of',\n",
       "  u'each',\n",
       "  u'person.',\n",
       "  u'The',\n",
       "  u'commitment',\n",
       "  u'to',\n",
       "  u'educating',\n",
       "  u'the',\n",
       "  u'whole',\n",
       "  u'person',\n",
       "  u'is',\n",
       "  u'evident',\n",
       "  u'in',\n",
       "  u\"USF's\",\n",
       "  u'strong',\n",
       "  u'core',\n",
       "  u'curriculum',\n",
       "  u'and',\n",
       "  u'its',\n",
       "  u'wide',\n",
       "  u'array',\n",
       "  u'of',\n",
       "  u'learning',\n",
       "  u'activities',\n",
       "  u'and',\n",
       "  u'experiences',\n",
       "  u'that',\n",
       "  u'bridges',\n",
       "  u'cultures',\n",
       "  u'and',\n",
       "  u'national',\n",
       "  u'boundaries.'],\n",
       " [],\n",
       " [u'4.',\n",
       "  u'\"Cura',\n",
       "  u'Personalis\"',\n",
       "  u'-',\n",
       "  u'\"Care',\n",
       "  u'of',\n",
       "  u'the',\n",
       "  u'Person\"',\n",
       "  u'was',\n",
       "  u'originally',\n",
       "  u'used',\n",
       "  u'to',\n",
       "  u'describe',\n",
       "  u'the',\n",
       "  u'responsibility',\n",
       "  u'of',\n",
       "  u'the',\n",
       "  u'Jesuit',\n",
       "  u'Superior',\n",
       "  u'to',\n",
       "  u'care',\n",
       "  u'for',\n",
       "  u'each',\n",
       "  u'man',\n",
       "  u'in',\n",
       "  u'the',\n",
       "  u'community',\n",
       "  u'with',\n",
       "  u'his',\n",
       "  u'unique',\n",
       "  u'gifts,',\n",
       "  u'challenges,',\n",
       "  u'needs,',\n",
       "  u'and',\n",
       "  u'possibilities.',\n",
       "  u'This',\n",
       "  u'value',\n",
       "  u'now',\n",
       "  u'is',\n",
       "  u'applied',\n",
       "  u'more',\n",
       "  u'broadly',\n",
       "  u'to',\n",
       "  u'include',\n",
       "  u'the',\n",
       "  u'relationship',\n",
       "  u'between',\n",
       "  u'educators',\n",
       "  u'and',\n",
       "  u'students,',\n",
       "  u'as',\n",
       "  u'well',\n",
       "  u'as',\n",
       "  u'the',\n",
       "  u'professional',\n",
       "  u'relationships',\n",
       "  u'among',\n",
       "  u'all',\n",
       "  u'those',\n",
       "  u'who',\n",
       "  u'work',\n",
       "  u'in',\n",
       "  u'the',\n",
       "  u'University.',\n",
       "  u'\"Cura',\n",
       "  u'Personalis\"',\n",
       "  u'suggests',\n",
       "  u'individualized',\n",
       "  u'attention',\n",
       "  u'to',\n",
       "  u'the',\n",
       "  u'needs',\n",
       "  u'of',\n",
       "  u'the',\n",
       "  u'other,',\n",
       "  u'distinct',\n",
       "  u'respect',\n",
       "  u'for',\n",
       "  u'his',\n",
       "  u'or',\n",
       "  u'her',\n",
       "  u'unique',\n",
       "  u'circumstances',\n",
       "  u'and',\n",
       "  u'concerns,',\n",
       "  u'and',\n",
       "  u'an',\n",
       "  u'appropriate',\n",
       "  u'appreciation',\n",
       "  u'for',\n",
       "  u'his',\n",
       "  u'or',\n",
       "  u'her',\n",
       "  u'particular',\n",
       "  u'gifts',\n",
       "  u'and',\n",
       "  u'insights.'],\n",
       " [],\n",
       " [u'5.',\n",
       "  u'Women',\n",
       "  u'and',\n",
       "  u'Men',\n",
       "  u'for',\n",
       "  u'Others',\n",
       "  u'-',\n",
       "  u'Fr.',\n",
       "  u'Pedro',\n",
       "  u'Arrupe,',\n",
       "  u'S.J.,',\n",
       "  u'Superior',\n",
       "  u'General',\n",
       "  u'of',\n",
       "  u'the',\n",
       "  u'Society',\n",
       "  u'of',\n",
       "  u'Jesuis',\n",
       "  u'from',\n",
       "  u'1965',\n",
       "  u'to',\n",
       "  u'1981,',\n",
       "  u'employed',\n",
       "  u'the',\n",
       "  u'phrase',\n",
       "  u'\"Men',\n",
       "  u'for',\n",
       "  u'Others\"',\n",
       "  u'in',\n",
       "  u'a',\n",
       "  u'notable',\n",
       "  u'1973',\n",
       "  u'presentation',\n",
       "  u'in',\n",
       "  u'Valencia,',\n",
       "  u'Spain.',\n",
       "  u'Father',\n",
       "  u'Arrupe',\n",
       "  u'provocatively',\n",
       "  u'challenged',\n",
       "  u'the',\n",
       "  u'alumni',\n",
       "  u'of',\n",
       "  u'Jesuit',\n",
       "  u'schools',\n",
       "  u'and',\n",
       "  u'universities',\n",
       "  u'to',\n",
       "  u'be',\n",
       "  u'engaged',\n",
       "  u'in',\n",
       "  u'the',\n",
       "  u'struggle',\n",
       "  u'for',\n",
       "  u'justice',\n",
       "  u'to',\n",
       "  u'protect',\n",
       "  u'the',\n",
       "  u'needs',\n",
       "  u'of',\n",
       "  u'those',\n",
       "  u'people',\n",
       "  u'who',\n",
       "  u'are',\n",
       "  u'most',\n",
       "  u'vulnerable.',\n",
       "  u'Today',\n",
       "  u'this',\n",
       "  u'phrase',\n",
       "  u'has',\n",
       "  u'become',\n",
       "  u'more',\n",
       "  u'inclusive',\n",
       "  u'and',\n",
       "  u'its',\n",
       "  u'spirit',\n",
       "  u'is',\n",
       "  u'evidenced',\n",
       "  u'in',\n",
       "  u\"USF's\",\n",
       "  u'promotion',\n",
       "  u'of',\n",
       "  u'service',\n",
       "  u'projects;',\n",
       "  u'justice',\n",
       "  u'learning',\n",
       "  u'through',\n",
       "  u'immersion;',\n",
       "  u'and',\n",
       "  u'its',\n",
       "  u'challenge',\n",
       "  u'to',\n",
       "  u'students,',\n",
       "  u'faculty,',\n",
       "  u'staff,',\n",
       "  u'and',\n",
       "  u'alumni',\n",
       "  u'to',\n",
       "  u'\"Change',\n",
       "  u'the',\n",
       "  u'World',\n",
       "  u'from',\n",
       "  u'Here.\"'],\n",
       " [],\n",
       " [u'6.',\n",
       "  u'Interreligious',\n",
       "  u'Understanding',\n",
       "  u'-',\n",
       "  u'Reflecting',\n",
       "  u'themes',\n",
       "  u'from',\n",
       "  u'the',\n",
       "  u'Second',\n",
       "  u'Vatican',\n",
       "  u'Council,',\n",
       "  u'the',\n",
       "  u'34th',\n",
       "  u'General',\n",
       "  u'Congregation',\n",
       "  u'of',\n",
       "  u'the',\n",
       "  u'Society',\n",
       "  u'of',\n",
       "  u'Jesus',\n",
       "  u'made',\n",
       "  u'a',\n",
       "  u'significant',\n",
       "  u'commitment',\n",
       "  u'to',\n",
       "  u'ecumenical',\n",
       "  u'and',\n",
       "  u'interreligious',\n",
       "  u'engagement',\n",
       "  u'and',\n",
       "  u'understanding.',\n",
       "  u'The',\n",
       "  u'USF',\n",
       "  u'community',\n",
       "  u'comprises',\n",
       "  u'a',\n",
       "  u'wide',\n",
       "  u'variety',\n",
       "  u'of',\n",
       "  u'religious',\n",
       "  u'traditions;',\n",
       "  u'and',\n",
       "  u'through',\n",
       "  u'its',\n",
       "  u'University',\n",
       "  u'Ministry,',\n",
       "  u'it',\n",
       "  u'encourages',\n",
       "  u'people',\n",
       "  u'of',\n",
       "  u'different',\n",
       "  u'faiths-and',\n",
       "  u'no',\n",
       "  u'faith-to',\n",
       "  u'create',\n",
       "  u'an',\n",
       "  u'inclusive',\n",
       "  u'community',\n",
       "  u'that',\n",
       "  u'is',\n",
       "  u'reflective',\n",
       "  u'and',\n",
       "  u'authentic',\n",
       "  u'in',\n",
       "  u'its',\n",
       "  u'search',\n",
       "  u'for',\n",
       "  u'meaning',\n",
       "  u'and',\n",
       "  u'truth.'],\n",
       " [],\n",
       " [u'7.',\n",
       "  u'Community',\n",
       "  u'in',\n",
       "  u'Diversity',\n",
       "  u'-',\n",
       "  u'As',\n",
       "  u'a',\n",
       "  u'Catholic',\n",
       "  u'and',\n",
       "  u'Jesuit',\n",
       "  u'University,',\n",
       "  u'USF',\n",
       "  u'welcomes',\n",
       "  u'and',\n",
       "  u'sustains',\n",
       "  u'the',\n",
       "  u'rich',\n",
       "  u'diversity',\n",
       "  u'among',\n",
       "  u'the',\n",
       "  u'students,',\n",
       "  u'faculty,',\n",
       "  u'and',\n",
       "  u'staff',\n",
       "  u'that',\n",
       "  u'comprises',\n",
       "  u'its',\n",
       "  u'community.',\n",
       "  u'The',\n",
       "  u'USF',\n",
       "  u'community',\n",
       "  u'continues',\n",
       "  u'to',\n",
       "  u'be',\n",
       "  u'enriched',\n",
       "  u'by',\n",
       "  u'people',\n",
       "  u'of',\n",
       "  u'different',\n",
       "  u'abilities,',\n",
       "  u'ages,',\n",
       "  u'colors,',\n",
       "  u'creeds,',\n",
       "  u'faiths,',\n",
       "  u'philosophies,',\n",
       "  u'gender',\n",
       "  u'orientations,',\n",
       "  u'cultures,',\n",
       "  u'nationalities,',\n",
       "  u'family',\n",
       "  u'backgrounds,',\n",
       "  u'and',\n",
       "  u'educational',\n",
       "  u'experiences.',\n",
       "  u'U.S.',\n",
       "  u'News',\n",
       "  u'and',\n",
       "  u'World',\n",
       "  u'Report',\n",
       "  u'ranks',\n",
       "  u'USF',\n",
       "  u'among',\n",
       "  u'the',\n",
       "  u\"nation's\",\n",
       "  u'top',\n",
       "  u'30',\n",
       "  u'university',\n",
       "  u'for',\n",
       "  u'student',\n",
       "  u'ethnic',\n",
       "  u'diversity',\n",
       "  u'and',\n",
       "  u'international',\n",
       "  u'student',\n",
       "  u'enrollment.'],\n",
       " [u'***********************************************************************************']]"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example 1-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "words = lines.flatMap(lambda line : line.split(\" \"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'=',\n",
       " u'Ignatian',\n",
       " u'Values',\n",
       " u'=',\n",
       " u'The',\n",
       " u'University',\n",
       " u'of',\n",
       " u'San',\n",
       " u'Francisco',\n",
       " u'enjoys',\n",
       " u'a',\n",
       " u'distinguished',\n",
       " u'heritage',\n",
       " u'and',\n",
       " u'Jesuit',\n",
       " u'tradition.',\n",
       " u'',\n",
       " u'At',\n",
       " u'the',\n",
       " u'core',\n",
       " u'of',\n",
       " u'this',\n",
       " u'tradition',\n",
       " u'are',\n",
       " u'transcendent',\n",
       " u'values,',\n",
       " u'including',\n",
       " u'the',\n",
       " u'integration',\n",
       " u'of',\n",
       " u'learning,',\n",
       " u'faith',\n",
       " u'and',\n",
       " u'service;',\n",
       " u'care',\n",
       " u'for',\n",
       " u'the',\n",
       " u'whole',\n",
       " u'person;',\n",
       " u'character',\n",
       " u'and',\n",
       " u'conviction;',\n",
       " u'religious',\n",
       " u'truth',\n",
       " u'and',\n",
       " u'interfaith',\n",
       " u'understanding;',\n",
       " u'and',\n",
       " u'a',\n",
       " u'commitment',\n",
       " u'to',\n",
       " u'building',\n",
       " u'a',\n",
       " u'more',\n",
       " u'just',\n",
       " u'world.',\n",
       " u'',\n",
       " u'The',\n",
       " u'key',\n",
       " u'values',\n",
       " u'of',\n",
       " u'this',\n",
       " u'Jesuit',\n",
       " u'tradition',\n",
       " u'are',\n",
       " u'as',\n",
       " u'follows:',\n",
       " u'***********************************************************************************',\n",
       " u'1.',\n",
       " u'Contemplative',\n",
       " u'in',\n",
       " u'Action',\n",
       " u'-',\n",
       " u'St.',\n",
       " u'Ignatius',\n",
       " u'Loyola',\n",
       " u'believed',\n",
       " u'that',\n",
       " u'prayer',\n",
       " u'and',\n",
       " u'reflectivity',\n",
       " u'should',\n",
       " u'so',\n",
       " u'guide',\n",
       " u'our',\n",
       " u'choices',\n",
       " u'and',\n",
       " u'actions',\n",
       " u'that',\n",
       " u'our',\n",
       " u'activity',\n",
       " u'itself',\n",
       " u'becomes',\n",
       " u'a',\n",
       " u'way',\n",
       " u'of',\n",
       " u'entering',\n",
       " u'into',\n",
       " u'union',\n",
       " u'with',\n",
       " u'and',\n",
       " u'praising',\n",
       " u'God.',\n",
       " u'',\n",
       " u'Being',\n",
       " u'a',\n",
       " u'contemplative',\n",
       " u'in',\n",
       " u'action',\n",
       " u'also',\n",
       " u'means',\n",
       " u'seeing',\n",
       " u'beyond',\n",
       " u'the',\n",
       " u'superficial',\n",
       " u'in',\n",
       " u'life',\n",
       " u'to',\n",
       " u'appreciate',\n",
       " u'the',\n",
       " u'mystery,',\n",
       " u'beauty,',\n",
       " u'and',\n",
       " u'sacredness',\n",
       " u'of',\n",
       " u'all',\n",
       " u'life.',\n",
       " u'',\n",
       " u'It',\n",
       " u'is',\n",
       " u'a',\n",
       " u'means',\n",
       " u'of',\n",
       " u'seeing',\n",
       " u'God',\n",
       " u'in',\n",
       " u'all',\n",
       " u'things',\n",
       " u'and',\n",
       " u'in',\n",
       " u'everyone.',\n",
       " u'',\n",
       " u'Contemplation',\n",
       " u'is',\n",
       " u'a',\n",
       " u'critical',\n",
       " u'dimension',\n",
       " u'of',\n",
       " u'the',\n",
       " u'spiritual',\n",
       " u'life',\n",
       " u'and',\n",
       " u'it',\n",
       " u'is',\n",
       " u'reflected',\n",
       " u'in',\n",
       " u\"USF's\",\n",
       " u'commitment',\n",
       " u'to',\n",
       " u'prayer',\n",
       " u'and',\n",
       " u'spiritual',\n",
       " u'growth.',\n",
       " u'',\n",
       " u'Analogously,',\n",
       " u'in',\n",
       " u'the',\n",
       " u'academic',\n",
       " u'life,',\n",
       " u'a',\n",
       " u'spirit',\n",
       " u'of',\n",
       " u'reflectivity',\n",
       " u'is',\n",
       " u'a',\n",
       " u'critical',\n",
       " u'aspect',\n",
       " u'of',\n",
       " u'intellectual',\n",
       " u'inquiry.',\n",
       " u'',\n",
       " u'2.',\n",
       " u'Academic',\n",
       " u'Excellence',\n",
       " u'-',\n",
       " u'In',\n",
       " u'1547,',\n",
       " u'the',\n",
       " u'first',\n",
       " u'Jesuits',\n",
       " u'were',\n",
       " u'invited',\n",
       " u'to',\n",
       " u'begin',\n",
       " u'a',\n",
       " u'college',\n",
       " u'in',\n",
       " u'Messina,',\n",
       " u'Italy,',\n",
       " u'so',\n",
       " u'that',\n",
       " u'the',\n",
       " u'young',\n",
       " u'men',\n",
       " u'of',\n",
       " u'that',\n",
       " u'town',\n",
       " u'could',\n",
       " u'receive',\n",
       " u'the',\n",
       " u'same',\n",
       " u'quality',\n",
       " u'of',\n",
       " u'education',\n",
       " u'that',\n",
       " u'the',\n",
       " u'early',\n",
       " u'Jesuits',\n",
       " u'promoted',\n",
       " u'in',\n",
       " u'training',\n",
       " u'their',\n",
       " u'own',\n",
       " u'men.',\n",
       " u'',\n",
       " u'USF',\n",
       " u'is',\n",
       " u'a',\n",
       " u'descendant',\n",
       " u'of',\n",
       " u'this',\n",
       " u'original',\n",
       " u'Jesuit',\n",
       " u'commitment',\n",
       " u'to',\n",
       " u'education.',\n",
       " u'',\n",
       " u'Academic',\n",
       " u'excellence',\n",
       " u'describes',\n",
       " u'the',\n",
       " u'great',\n",
       " u'importance',\n",
       " u'that',\n",
       " u'Jesuits',\n",
       " u'have',\n",
       " u'placed',\n",
       " u'on',\n",
       " u'the',\n",
       " u'life',\n",
       " u'of',\n",
       " u'the',\n",
       " u'mind',\n",
       " u'as',\n",
       " u'a',\n",
       " u'means',\n",
       " u'for',\n",
       " u'uncovering',\n",
       " u'truth',\n",
       " u'and',\n",
       " u'discovering',\n",
       " u'meaning.',\n",
       " u'',\n",
       " u'It',\n",
       " u'is',\n",
       " u'this',\n",
       " u'tradition',\n",
       " u'that',\n",
       " u'imbues',\n",
       " u'the',\n",
       " u'other',\n",
       " u'27',\n",
       " u'Jesuit',\n",
       " u'universities',\n",
       " u'and',\n",
       " u'colleges,',\n",
       " u'as',\n",
       " u'well',\n",
       " u'as',\n",
       " u'the',\n",
       " u'Jesuit',\n",
       " u'high',\n",
       " u'schools',\n",
       " u'and',\n",
       " u'middle',\n",
       " u'schools',\n",
       " u'in',\n",
       " u'the',\n",
       " u'US.',\n",
       " u'',\n",
       " u'3.',\n",
       " u'Educating',\n",
       " u'the',\n",
       " u'Whole',\n",
       " u'Person',\n",
       " u'-',\n",
       " u'St.',\n",
       " u'Ignatius',\n",
       " u'believed',\n",
       " u'that',\n",
       " u'God',\n",
       " u'could',\n",
       " u'be',\n",
       " u'discovered',\n",
       " u'in',\n",
       " u'every',\n",
       " u'human',\n",
       " u'endeavor,',\n",
       " u'in',\n",
       " u'every',\n",
       " u'facet',\n",
       " u'of',\n",
       " u'learning',\n",
       " u'and',\n",
       " u'experience,',\n",
       " u'and',\n",
       " u'in',\n",
       " u'every',\n",
       " u'field',\n",
       " u'of',\n",
       " u'study.',\n",
       " u'',\n",
       " u'Consequently,',\n",
       " u'he',\n",
       " u'promoted',\n",
       " u'the',\n",
       " u'development',\n",
       " u'of',\n",
       " u'the',\n",
       " u'spiritual,',\n",
       " u'intellectual,',\n",
       " u'artistic,',\n",
       " u'social,',\n",
       " u'and',\n",
       " u'physical',\n",
       " u'aspects',\n",
       " u'of',\n",
       " u'each',\n",
       " u'person.',\n",
       " u'',\n",
       " u'The',\n",
       " u'commitment',\n",
       " u'to',\n",
       " u'educating',\n",
       " u'the',\n",
       " u'whole',\n",
       " u'person',\n",
       " u'is',\n",
       " u'evident',\n",
       " u'in',\n",
       " u\"USF's\",\n",
       " u'strong',\n",
       " u'core',\n",
       " u'curriculum',\n",
       " u'and',\n",
       " u'its',\n",
       " u'wide',\n",
       " u'array',\n",
       " u'of',\n",
       " u'learning',\n",
       " u'activities',\n",
       " u'and',\n",
       " u'experiences',\n",
       " u'that',\n",
       " u'bridges',\n",
       " u'cultures',\n",
       " u'and',\n",
       " u'national',\n",
       " u'boundaries.',\n",
       " u'',\n",
       " u'4.',\n",
       " u'\"Cura',\n",
       " u'Personalis\"',\n",
       " u'-',\n",
       " u'\"Care',\n",
       " u'of',\n",
       " u'the',\n",
       " u'Person\"',\n",
       " u'was',\n",
       " u'originally',\n",
       " u'used',\n",
       " u'to',\n",
       " u'describe',\n",
       " u'the',\n",
       " u'responsibility',\n",
       " u'of',\n",
       " u'the',\n",
       " u'Jesuit',\n",
       " u'Superior',\n",
       " u'to',\n",
       " u'care',\n",
       " u'for',\n",
       " u'each',\n",
       " u'man',\n",
       " u'in',\n",
       " u'the',\n",
       " u'community',\n",
       " u'with',\n",
       " u'his',\n",
       " u'unique',\n",
       " u'gifts,',\n",
       " u'challenges,',\n",
       " u'needs,',\n",
       " u'and',\n",
       " u'possibilities.',\n",
       " u'',\n",
       " u'This',\n",
       " u'value',\n",
       " u'now',\n",
       " u'is',\n",
       " u'applied',\n",
       " u'more',\n",
       " u'broadly',\n",
       " u'to',\n",
       " u'include',\n",
       " u'the',\n",
       " u'relationship',\n",
       " u'between',\n",
       " u'educators',\n",
       " u'and',\n",
       " u'students,',\n",
       " u'as',\n",
       " u'well',\n",
       " u'as',\n",
       " u'the',\n",
       " u'professional',\n",
       " u'relationships',\n",
       " u'among',\n",
       " u'all',\n",
       " u'those',\n",
       " u'who',\n",
       " u'work',\n",
       " u'in',\n",
       " u'the',\n",
       " u'University.',\n",
       " u'',\n",
       " u'\"Cura',\n",
       " u'Personalis\"',\n",
       " u'suggests',\n",
       " u'individualized',\n",
       " u'attention',\n",
       " u'to',\n",
       " u'the',\n",
       " u'needs',\n",
       " u'of',\n",
       " u'the',\n",
       " u'other,',\n",
       " u'distinct',\n",
       " u'respect',\n",
       " u'for',\n",
       " u'his',\n",
       " u'or',\n",
       " u'her',\n",
       " u'unique',\n",
       " u'circumstances',\n",
       " u'and',\n",
       " u'concerns,',\n",
       " u'and',\n",
       " u'an',\n",
       " u'appropriate',\n",
       " u'appreciation',\n",
       " u'for',\n",
       " u'his',\n",
       " u'or',\n",
       " u'her',\n",
       " u'particular',\n",
       " u'gifts',\n",
       " u'and',\n",
       " u'insights.',\n",
       " u'',\n",
       " u'5.',\n",
       " u'Women',\n",
       " u'and',\n",
       " u'Men',\n",
       " u'for',\n",
       " u'Others',\n",
       " u'-',\n",
       " u'Fr.',\n",
       " u'Pedro',\n",
       " u'Arrupe,',\n",
       " u'S.J.,',\n",
       " u'Superior',\n",
       " u'General',\n",
       " u'of',\n",
       " u'the',\n",
       " u'Society',\n",
       " u'of',\n",
       " u'Jesuis',\n",
       " u'from',\n",
       " u'1965',\n",
       " u'to',\n",
       " u'1981,',\n",
       " u'employed',\n",
       " u'the',\n",
       " u'phrase',\n",
       " u'\"Men',\n",
       " u'for',\n",
       " u'Others\"',\n",
       " u'in',\n",
       " u'a',\n",
       " u'notable',\n",
       " u'1973',\n",
       " u'presentation',\n",
       " u'in',\n",
       " u'Valencia,',\n",
       " u'Spain.',\n",
       " u'',\n",
       " u'Father',\n",
       " u'Arrupe',\n",
       " u'provocatively',\n",
       " u'challenged',\n",
       " u'the',\n",
       " u'alumni',\n",
       " u'of',\n",
       " u'Jesuit',\n",
       " u'schools',\n",
       " u'and',\n",
       " u'universities',\n",
       " u'to',\n",
       " u'be',\n",
       " u'engaged',\n",
       " u'in',\n",
       " u'the',\n",
       " u'struggle',\n",
       " u'for',\n",
       " u'justice',\n",
       " u'to',\n",
       " u'protect',\n",
       " u'the',\n",
       " u'needs',\n",
       " u'of',\n",
       " u'those',\n",
       " u'people',\n",
       " u'who',\n",
       " u'are',\n",
       " u'most',\n",
       " u'vulnerable.',\n",
       " u'',\n",
       " u'Today',\n",
       " u'this',\n",
       " u'phrase',\n",
       " u'has',\n",
       " u'become',\n",
       " u'more',\n",
       " u'inclusive',\n",
       " u'and',\n",
       " u'its',\n",
       " u'spirit',\n",
       " u'is',\n",
       " u'evidenced',\n",
       " u'in',\n",
       " u\"USF's\",\n",
       " u'promotion',\n",
       " u'of',\n",
       " u'service',\n",
       " u'projects;',\n",
       " u'justice',\n",
       " u'learning',\n",
       " u'through',\n",
       " u'immersion;',\n",
       " u'and',\n",
       " u'its',\n",
       " u'challenge',\n",
       " u'to',\n",
       " u'students,',\n",
       " u'faculty,',\n",
       " u'staff,',\n",
       " u'and',\n",
       " u'alumni',\n",
       " u'to',\n",
       " u'\"Change',\n",
       " u'the',\n",
       " u'World',\n",
       " u'from',\n",
       " u'Here.\"',\n",
       " u'',\n",
       " u'6.',\n",
       " u'Interreligious',\n",
       " u'Understanding',\n",
       " u'-',\n",
       " u'Reflecting',\n",
       " u'themes',\n",
       " u'from',\n",
       " u'the',\n",
       " u'Second',\n",
       " u'Vatican',\n",
       " u'Council,',\n",
       " u'the',\n",
       " u'34th',\n",
       " u'General',\n",
       " u'Congregation',\n",
       " u'of',\n",
       " u'the',\n",
       " u'Society',\n",
       " u'of',\n",
       " u'Jesus',\n",
       " u'made',\n",
       " u'a',\n",
       " u'significant',\n",
       " u'commitment',\n",
       " u'to',\n",
       " u'ecumenical',\n",
       " u'and',\n",
       " u'interreligious',\n",
       " u'engagement',\n",
       " u'and',\n",
       " u'understanding.',\n",
       " u'',\n",
       " u'The',\n",
       " u'USF',\n",
       " u'community',\n",
       " u'comprises',\n",
       " u'a',\n",
       " u'wide',\n",
       " u'variety',\n",
       " u'of',\n",
       " u'religious',\n",
       " u'traditions;',\n",
       " u'and',\n",
       " u'through',\n",
       " u'its',\n",
       " u'University',\n",
       " u'Ministry,',\n",
       " u'it',\n",
       " u'encourages',\n",
       " u'people',\n",
       " u'of',\n",
       " u'different',\n",
       " u'faiths-and',\n",
       " u'no',\n",
       " u'faith-to',\n",
       " u'create',\n",
       " u'an',\n",
       " u'inclusive',\n",
       " u'community',\n",
       " u'that',\n",
       " u'is',\n",
       " u'reflective',\n",
       " u'and',\n",
       " u'authentic',\n",
       " u'in',\n",
       " u'its',\n",
       " u'search',\n",
       " u'for',\n",
       " u'meaning',\n",
       " u'and',\n",
       " u'truth.',\n",
       " u'',\n",
       " u'7.',\n",
       " u'Community',\n",
       " u'in',\n",
       " u'Diversity',\n",
       " u'-',\n",
       " u'As',\n",
       " u'a',\n",
       " u'Catholic',\n",
       " u'and',\n",
       " u'Jesuit',\n",
       " u'University,',\n",
       " u'USF',\n",
       " u'welcomes',\n",
       " u'and',\n",
       " u'sustains',\n",
       " u'the',\n",
       " u'rich',\n",
       " u'diversity',\n",
       " u'among',\n",
       " u'the',\n",
       " u'students,',\n",
       " u'faculty,',\n",
       " u'and',\n",
       " u'staff',\n",
       " u'that',\n",
       " u'comprises',\n",
       " u'its',\n",
       " u'community.',\n",
       " u'',\n",
       " u'The',\n",
       " u'USF',\n",
       " u'community',\n",
       " u'continues',\n",
       " u'to',\n",
       " u'be',\n",
       " u'enriched',\n",
       " u'by',\n",
       " u'people',\n",
       " u'of',\n",
       " u'different',\n",
       " u'abilities,',\n",
       " u'ages,',\n",
       " u'colors,',\n",
       " u'creeds,',\n",
       " u'faiths,',\n",
       " u'philosophies,',\n",
       " u'gender',\n",
       " u'orientations,',\n",
       " u'cultures,',\n",
       " u'nationalities,',\n",
       " u'family',\n",
       " u'backgrounds,',\n",
       " u'and',\n",
       " u'educational',\n",
       " u'experiences.',\n",
       " u'',\n",
       " u'U.S.',\n",
       " u'News',\n",
       " u'and',\n",
       " u'World',\n",
       " u'Report',\n",
       " u'ranks',\n",
       " u'USF',\n",
       " u'among',\n",
       " u'the',\n",
       " u\"nation's\",\n",
       " u'top',\n",
       " u'30',\n",
       " u'university',\n",
       " u'for',\n",
       " u'student',\n",
       " u'ethnic',\n",
       " u'diversity',\n",
       " u'and',\n",
       " u'international',\n",
       " u'student',\n",
       " u'enrollment.',\n",
       " u'***********************************************************************************']"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example 1-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "words = words.filter(lambda lines : lines is not u'')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'=',\n",
       " u'Ignatian',\n",
       " u'Values',\n",
       " u'=',\n",
       " u'The',\n",
       " u'University',\n",
       " u'of',\n",
       " u'San',\n",
       " u'Francisco',\n",
       " u'enjoys',\n",
       " u'a',\n",
       " u'distinguished',\n",
       " u'heritage',\n",
       " u'and',\n",
       " u'Jesuit',\n",
       " u'tradition.',\n",
       " u'At',\n",
       " u'the',\n",
       " u'core',\n",
       " u'of',\n",
       " u'this',\n",
       " u'tradition',\n",
       " u'are',\n",
       " u'transcendent',\n",
       " u'values,',\n",
       " u'including',\n",
       " u'the',\n",
       " u'integration',\n",
       " u'of',\n",
       " u'learning,',\n",
       " u'faith',\n",
       " u'and',\n",
       " u'service;',\n",
       " u'care',\n",
       " u'for',\n",
       " u'the',\n",
       " u'whole',\n",
       " u'person;',\n",
       " u'character',\n",
       " u'and',\n",
       " u'conviction;',\n",
       " u'religious',\n",
       " u'truth',\n",
       " u'and',\n",
       " u'interfaith',\n",
       " u'understanding;',\n",
       " u'and',\n",
       " u'a',\n",
       " u'commitment',\n",
       " u'to',\n",
       " u'building',\n",
       " u'a',\n",
       " u'more',\n",
       " u'just',\n",
       " u'world.',\n",
       " u'The',\n",
       " u'key',\n",
       " u'values',\n",
       " u'of',\n",
       " u'this',\n",
       " u'Jesuit',\n",
       " u'tradition',\n",
       " u'are',\n",
       " u'as',\n",
       " u'follows:',\n",
       " u'***********************************************************************************',\n",
       " u'1.',\n",
       " u'Contemplative',\n",
       " u'in',\n",
       " u'Action',\n",
       " u'-',\n",
       " u'St.',\n",
       " u'Ignatius',\n",
       " u'Loyola',\n",
       " u'believed',\n",
       " u'that',\n",
       " u'prayer',\n",
       " u'and',\n",
       " u'reflectivity',\n",
       " u'should',\n",
       " u'so',\n",
       " u'guide',\n",
       " u'our',\n",
       " u'choices',\n",
       " u'and',\n",
       " u'actions',\n",
       " u'that',\n",
       " u'our',\n",
       " u'activity',\n",
       " u'itself',\n",
       " u'becomes',\n",
       " u'a',\n",
       " u'way',\n",
       " u'of',\n",
       " u'entering',\n",
       " u'into',\n",
       " u'union',\n",
       " u'with',\n",
       " u'and',\n",
       " u'praising',\n",
       " u'God.',\n",
       " u'Being',\n",
       " u'a',\n",
       " u'contemplative',\n",
       " u'in',\n",
       " u'action',\n",
       " u'also',\n",
       " u'means',\n",
       " u'seeing',\n",
       " u'beyond',\n",
       " u'the',\n",
       " u'superficial',\n",
       " u'in',\n",
       " u'life',\n",
       " u'to',\n",
       " u'appreciate',\n",
       " u'the',\n",
       " u'mystery,',\n",
       " u'beauty,',\n",
       " u'and',\n",
       " u'sacredness',\n",
       " u'of',\n",
       " u'all',\n",
       " u'life.',\n",
       " u'It',\n",
       " u'is',\n",
       " u'a',\n",
       " u'means',\n",
       " u'of',\n",
       " u'seeing',\n",
       " u'God',\n",
       " u'in',\n",
       " u'all',\n",
       " u'things',\n",
       " u'and',\n",
       " u'in',\n",
       " u'everyone.',\n",
       " u'Contemplation',\n",
       " u'is',\n",
       " u'a',\n",
       " u'critical',\n",
       " u'dimension',\n",
       " u'of',\n",
       " u'the',\n",
       " u'spiritual',\n",
       " u'life',\n",
       " u'and',\n",
       " u'it',\n",
       " u'is',\n",
       " u'reflected',\n",
       " u'in',\n",
       " u\"USF's\",\n",
       " u'commitment',\n",
       " u'to',\n",
       " u'prayer',\n",
       " u'and',\n",
       " u'spiritual',\n",
       " u'growth.',\n",
       " u'Analogously,',\n",
       " u'in',\n",
       " u'the',\n",
       " u'academic',\n",
       " u'life,',\n",
       " u'a',\n",
       " u'spirit',\n",
       " u'of',\n",
       " u'reflectivity',\n",
       " u'is',\n",
       " u'a',\n",
       " u'critical',\n",
       " u'aspect',\n",
       " u'of',\n",
       " u'intellectual',\n",
       " u'inquiry.',\n",
       " u'2.',\n",
       " u'Academic',\n",
       " u'Excellence',\n",
       " u'-',\n",
       " u'In',\n",
       " u'1547,',\n",
       " u'the',\n",
       " u'first',\n",
       " u'Jesuits',\n",
       " u'were',\n",
       " u'invited',\n",
       " u'to',\n",
       " u'begin',\n",
       " u'a',\n",
       " u'college',\n",
       " u'in',\n",
       " u'Messina,',\n",
       " u'Italy,',\n",
       " u'so',\n",
       " u'that',\n",
       " u'the',\n",
       " u'young',\n",
       " u'men',\n",
       " u'of',\n",
       " u'that',\n",
       " u'town',\n",
       " u'could',\n",
       " u'receive',\n",
       " u'the',\n",
       " u'same',\n",
       " u'quality',\n",
       " u'of',\n",
       " u'education',\n",
       " u'that',\n",
       " u'the',\n",
       " u'early',\n",
       " u'Jesuits',\n",
       " u'promoted',\n",
       " u'in',\n",
       " u'training',\n",
       " u'their',\n",
       " u'own',\n",
       " u'men.',\n",
       " u'USF',\n",
       " u'is',\n",
       " u'a',\n",
       " u'descendant',\n",
       " u'of',\n",
       " u'this',\n",
       " u'original',\n",
       " u'Jesuit',\n",
       " u'commitment',\n",
       " u'to',\n",
       " u'education.',\n",
       " u'Academic',\n",
       " u'excellence',\n",
       " u'describes',\n",
       " u'the',\n",
       " u'great',\n",
       " u'importance',\n",
       " u'that',\n",
       " u'Jesuits',\n",
       " u'have',\n",
       " u'placed',\n",
       " u'on',\n",
       " u'the',\n",
       " u'life',\n",
       " u'of',\n",
       " u'the',\n",
       " u'mind',\n",
       " u'as',\n",
       " u'a',\n",
       " u'means',\n",
       " u'for',\n",
       " u'uncovering',\n",
       " u'truth',\n",
       " u'and',\n",
       " u'discovering',\n",
       " u'meaning.',\n",
       " u'It',\n",
       " u'is',\n",
       " u'this',\n",
       " u'tradition',\n",
       " u'that',\n",
       " u'imbues',\n",
       " u'the',\n",
       " u'other',\n",
       " u'27',\n",
       " u'Jesuit',\n",
       " u'universities',\n",
       " u'and',\n",
       " u'colleges,',\n",
       " u'as',\n",
       " u'well',\n",
       " u'as',\n",
       " u'the',\n",
       " u'Jesuit',\n",
       " u'high',\n",
       " u'schools',\n",
       " u'and',\n",
       " u'middle',\n",
       " u'schools',\n",
       " u'in',\n",
       " u'the',\n",
       " u'US.',\n",
       " u'3.',\n",
       " u'Educating',\n",
       " u'the',\n",
       " u'Whole',\n",
       " u'Person',\n",
       " u'-',\n",
       " u'St.',\n",
       " u'Ignatius',\n",
       " u'believed',\n",
       " u'that',\n",
       " u'God',\n",
       " u'could',\n",
       " u'be',\n",
       " u'discovered',\n",
       " u'in',\n",
       " u'every',\n",
       " u'human',\n",
       " u'endeavor,',\n",
       " u'in',\n",
       " u'every',\n",
       " u'facet',\n",
       " u'of',\n",
       " u'learning',\n",
       " u'and',\n",
       " u'experience,',\n",
       " u'and',\n",
       " u'in',\n",
       " u'every',\n",
       " u'field',\n",
       " u'of',\n",
       " u'study.',\n",
       " u'Consequently,',\n",
       " u'he',\n",
       " u'promoted',\n",
       " u'the',\n",
       " u'development',\n",
       " u'of',\n",
       " u'the',\n",
       " u'spiritual,',\n",
       " u'intellectual,',\n",
       " u'artistic,',\n",
       " u'social,',\n",
       " u'and',\n",
       " u'physical',\n",
       " u'aspects',\n",
       " u'of',\n",
       " u'each',\n",
       " u'person.',\n",
       " u'The',\n",
       " u'commitment',\n",
       " u'to',\n",
       " u'educating',\n",
       " u'the',\n",
       " u'whole',\n",
       " u'person',\n",
       " u'is',\n",
       " u'evident',\n",
       " u'in',\n",
       " u\"USF's\",\n",
       " u'strong',\n",
       " u'core',\n",
       " u'curriculum',\n",
       " u'and',\n",
       " u'its',\n",
       " u'wide',\n",
       " u'array',\n",
       " u'of',\n",
       " u'learning',\n",
       " u'activities',\n",
       " u'and',\n",
       " u'experiences',\n",
       " u'that',\n",
       " u'bridges',\n",
       " u'cultures',\n",
       " u'and',\n",
       " u'national',\n",
       " u'boundaries.',\n",
       " u'4.',\n",
       " u'\"Cura',\n",
       " u'Personalis\"',\n",
       " u'-',\n",
       " u'\"Care',\n",
       " u'of',\n",
       " u'the',\n",
       " u'Person\"',\n",
       " u'was',\n",
       " u'originally',\n",
       " u'used',\n",
       " u'to',\n",
       " u'describe',\n",
       " u'the',\n",
       " u'responsibility',\n",
       " u'of',\n",
       " u'the',\n",
       " u'Jesuit',\n",
       " u'Superior',\n",
       " u'to',\n",
       " u'care',\n",
       " u'for',\n",
       " u'each',\n",
       " u'man',\n",
       " u'in',\n",
       " u'the',\n",
       " u'community',\n",
       " u'with',\n",
       " u'his',\n",
       " u'unique',\n",
       " u'gifts,',\n",
       " u'challenges,',\n",
       " u'needs,',\n",
       " u'and',\n",
       " u'possibilities.',\n",
       " u'This',\n",
       " u'value',\n",
       " u'now',\n",
       " u'is',\n",
       " u'applied',\n",
       " u'more',\n",
       " u'broadly',\n",
       " u'to',\n",
       " u'include',\n",
       " u'the',\n",
       " u'relationship',\n",
       " u'between',\n",
       " u'educators',\n",
       " u'and',\n",
       " u'students,',\n",
       " u'as',\n",
       " u'well',\n",
       " u'as',\n",
       " u'the',\n",
       " u'professional',\n",
       " u'relationships',\n",
       " u'among',\n",
       " u'all',\n",
       " u'those',\n",
       " u'who',\n",
       " u'work',\n",
       " u'in',\n",
       " u'the',\n",
       " u'University.',\n",
       " u'\"Cura',\n",
       " u'Personalis\"',\n",
       " u'suggests',\n",
       " u'individualized',\n",
       " u'attention',\n",
       " u'to',\n",
       " u'the',\n",
       " u'needs',\n",
       " u'of',\n",
       " u'the',\n",
       " u'other,',\n",
       " u'distinct',\n",
       " u'respect',\n",
       " u'for',\n",
       " u'his',\n",
       " u'or',\n",
       " u'her',\n",
       " u'unique',\n",
       " u'circumstances',\n",
       " u'and',\n",
       " u'concerns,',\n",
       " u'and',\n",
       " u'an',\n",
       " u'appropriate',\n",
       " u'appreciation',\n",
       " u'for',\n",
       " u'his',\n",
       " u'or',\n",
       " u'her',\n",
       " u'particular',\n",
       " u'gifts',\n",
       " u'and',\n",
       " u'insights.',\n",
       " u'5.',\n",
       " u'Women',\n",
       " u'and',\n",
       " u'Men',\n",
       " u'for',\n",
       " u'Others',\n",
       " u'-',\n",
       " u'Fr.',\n",
       " u'Pedro',\n",
       " u'Arrupe,',\n",
       " u'S.J.,',\n",
       " u'Superior',\n",
       " u'General',\n",
       " u'of',\n",
       " u'the',\n",
       " u'Society',\n",
       " u'of',\n",
       " u'Jesuis',\n",
       " u'from',\n",
       " u'1965',\n",
       " u'to',\n",
       " u'1981,',\n",
       " u'employed',\n",
       " u'the',\n",
       " u'phrase',\n",
       " u'\"Men',\n",
       " u'for',\n",
       " u'Others\"',\n",
       " u'in',\n",
       " u'a',\n",
       " u'notable',\n",
       " u'1973',\n",
       " u'presentation',\n",
       " u'in',\n",
       " u'Valencia,',\n",
       " u'Spain.',\n",
       " u'Father',\n",
       " u'Arrupe',\n",
       " u'provocatively',\n",
       " u'challenged',\n",
       " u'the',\n",
       " u'alumni',\n",
       " u'of',\n",
       " u'Jesuit',\n",
       " u'schools',\n",
       " u'and',\n",
       " u'universities',\n",
       " u'to',\n",
       " u'be',\n",
       " u'engaged',\n",
       " u'in',\n",
       " u'the',\n",
       " u'struggle',\n",
       " u'for',\n",
       " u'justice',\n",
       " u'to',\n",
       " u'protect',\n",
       " u'the',\n",
       " u'needs',\n",
       " u'of',\n",
       " u'those',\n",
       " u'people',\n",
       " u'who',\n",
       " u'are',\n",
       " u'most',\n",
       " u'vulnerable.',\n",
       " u'Today',\n",
       " u'this',\n",
       " u'phrase',\n",
       " u'has',\n",
       " u'become',\n",
       " u'more',\n",
       " u'inclusive',\n",
       " u'and',\n",
       " u'its',\n",
       " u'spirit',\n",
       " u'is',\n",
       " u'evidenced',\n",
       " u'in',\n",
       " u\"USF's\",\n",
       " u'promotion',\n",
       " u'of',\n",
       " u'service',\n",
       " u'projects;',\n",
       " u'justice',\n",
       " u'learning',\n",
       " u'through',\n",
       " u'immersion;',\n",
       " u'and',\n",
       " u'its',\n",
       " u'challenge',\n",
       " u'to',\n",
       " u'students,',\n",
       " u'faculty,',\n",
       " u'staff,',\n",
       " u'and',\n",
       " u'alumni',\n",
       " u'to',\n",
       " u'\"Change',\n",
       " u'the',\n",
       " u'World',\n",
       " u'from',\n",
       " u'Here.\"',\n",
       " u'6.',\n",
       " u'Interreligious',\n",
       " u'Understanding',\n",
       " u'-',\n",
       " u'Reflecting',\n",
       " u'themes',\n",
       " u'from',\n",
       " u'the',\n",
       " u'Second',\n",
       " u'Vatican',\n",
       " u'Council,',\n",
       " u'the',\n",
       " u'34th',\n",
       " u'General',\n",
       " u'Congregation',\n",
       " u'of',\n",
       " u'the',\n",
       " u'Society',\n",
       " u'of',\n",
       " u'Jesus',\n",
       " u'made',\n",
       " u'a',\n",
       " u'significant',\n",
       " u'commitment',\n",
       " u'to',\n",
       " u'ecumenical',\n",
       " u'and',\n",
       " u'interreligious',\n",
       " u'engagement',\n",
       " u'and',\n",
       " u'understanding.',\n",
       " u'The',\n",
       " u'USF',\n",
       " u'community',\n",
       " u'comprises',\n",
       " u'a',\n",
       " u'wide',\n",
       " u'variety',\n",
       " u'of',\n",
       " u'religious',\n",
       " u'traditions;',\n",
       " u'and',\n",
       " u'through',\n",
       " u'its',\n",
       " u'University',\n",
       " u'Ministry,',\n",
       " u'it',\n",
       " u'encourages',\n",
       " u'people',\n",
       " u'of',\n",
       " u'different',\n",
       " u'faiths-and',\n",
       " u'no',\n",
       " u'faith-to',\n",
       " u'create',\n",
       " u'an',\n",
       " u'inclusive',\n",
       " u'community',\n",
       " u'that',\n",
       " u'is',\n",
       " u'reflective',\n",
       " u'and',\n",
       " u'authentic',\n",
       " u'in',\n",
       " u'its',\n",
       " u'search',\n",
       " u'for',\n",
       " u'meaning',\n",
       " u'and',\n",
       " u'truth.',\n",
       " u'7.',\n",
       " u'Community',\n",
       " u'in',\n",
       " u'Diversity',\n",
       " u'-',\n",
       " u'As',\n",
       " u'a',\n",
       " u'Catholic',\n",
       " u'and',\n",
       " u'Jesuit',\n",
       " u'University,',\n",
       " u'USF',\n",
       " u'welcomes',\n",
       " u'and',\n",
       " u'sustains',\n",
       " u'the',\n",
       " u'rich',\n",
       " u'diversity',\n",
       " u'among',\n",
       " u'the',\n",
       " u'students,',\n",
       " u'faculty,',\n",
       " u'and',\n",
       " u'staff',\n",
       " u'that',\n",
       " u'comprises',\n",
       " u'its',\n",
       " u'community.',\n",
       " u'The',\n",
       " u'USF',\n",
       " u'community',\n",
       " u'continues',\n",
       " u'to',\n",
       " u'be',\n",
       " u'enriched',\n",
       " u'by',\n",
       " u'people',\n",
       " u'of',\n",
       " u'different',\n",
       " u'abilities,',\n",
       " u'ages,',\n",
       " u'colors,',\n",
       " u'creeds,',\n",
       " u'faiths,',\n",
       " u'philosophies,',\n",
       " u'gender',\n",
       " u'orientations,',\n",
       " u'cultures,',\n",
       " u'nationalities,',\n",
       " u'family',\n",
       " u'backgrounds,',\n",
       " u'and',\n",
       " u'educational',\n",
       " u'experiences.',\n",
       " u'U.S.',\n",
       " u'News',\n",
       " u'and',\n",
       " u'World',\n",
       " u'Report',\n",
       " u'ranks',\n",
       " u'USF',\n",
       " u'among',\n",
       " u'the',\n",
       " u\"nation's\",\n",
       " u'top',\n",
       " u'30',\n",
       " u'university',\n",
       " u'for',\n",
       " u'student',\n",
       " u'ethnic',\n",
       " u'diversity',\n",
       " u'and',\n",
       " u'international',\n",
       " u'student',\n",
       " u'enrollment.',\n",
       " u'***********************************************************************************']"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "USF_words = words.filter(lambda lines : \"USF\" in lines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u\"USF's\", u'USF', u\"USF's\", u\"USF's\", u'USF', u'USF', u'USF', u'USF']"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "USF_words.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example 1-4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def split_function(partition):\n",
    "        word = str(list(partition)).split()\n",
    "        yield word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "words = lines.mapPartitions(split_function)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[\"[u'=\",\n",
       "  'Ignatian',\n",
       "  'Values',\n",
       "  \"=',\",\n",
       "  \"u'The\",\n",
       "  'University',\n",
       "  'of',\n",
       "  'San',\n",
       "  'Francisco',\n",
       "  'enjoys',\n",
       "  'a',\n",
       "  'distinguished',\n",
       "  'heritage',\n",
       "  'and',\n",
       "  'Jesuit',\n",
       "  'tradition.',\n",
       "  'At',\n",
       "  'the',\n",
       "  'core',\n",
       "  'of',\n",
       "  'this',\n",
       "  'tradition',\n",
       "  'are',\n",
       "  'transcendent',\n",
       "  'values,',\n",
       "  'including',\n",
       "  'the',\n",
       "  'integration',\n",
       "  'of',\n",
       "  'learning,',\n",
       "  'faith',\n",
       "  'and',\n",
       "  'service;',\n",
       "  'care',\n",
       "  'for',\n",
       "  'the',\n",
       "  'whole',\n",
       "  'person;',\n",
       "  'character',\n",
       "  'and',\n",
       "  'conviction;',\n",
       "  'religious',\n",
       "  'truth',\n",
       "  'and',\n",
       "  'interfaith',\n",
       "  'understanding;',\n",
       "  'and',\n",
       "  'a',\n",
       "  'commitment',\n",
       "  'to',\n",
       "  'building',\n",
       "  'a',\n",
       "  'more',\n",
       "  'just',\n",
       "  'world.',\n",
       "  'The',\n",
       "  'key',\n",
       "  'values',\n",
       "  'of',\n",
       "  'this',\n",
       "  'Jesuit',\n",
       "  'tradition',\n",
       "  'are',\n",
       "  'as',\n",
       "  \"follows:',\",\n",
       "  \"u'***********************************************************************************',\",\n",
       "  'u\"1.',\n",
       "  'Contemplative',\n",
       "  'in',\n",
       "  'Action',\n",
       "  '-',\n",
       "  'St.',\n",
       "  'Ignatius',\n",
       "  'Loyola',\n",
       "  'believed',\n",
       "  'that',\n",
       "  'prayer',\n",
       "  'and',\n",
       "  'reflectivity',\n",
       "  'should',\n",
       "  'so',\n",
       "  'guide',\n",
       "  'our',\n",
       "  'choices',\n",
       "  'and',\n",
       "  'actions',\n",
       "  'that',\n",
       "  'our',\n",
       "  'activity',\n",
       "  'itself',\n",
       "  'becomes',\n",
       "  'a',\n",
       "  'way',\n",
       "  'of',\n",
       "  'entering',\n",
       "  'into',\n",
       "  'union',\n",
       "  'with',\n",
       "  'and',\n",
       "  'praising',\n",
       "  'God.',\n",
       "  'Being',\n",
       "  'a',\n",
       "  'contemplative',\n",
       "  'in',\n",
       "  'action',\n",
       "  'also',\n",
       "  'means',\n",
       "  'seeing',\n",
       "  'beyond',\n",
       "  'the',\n",
       "  'superficial',\n",
       "  'in',\n",
       "  'life',\n",
       "  'to',\n",
       "  'appreciate',\n",
       "  'the',\n",
       "  'mystery,',\n",
       "  'beauty,',\n",
       "  'and',\n",
       "  'sacredness',\n",
       "  'of',\n",
       "  'all',\n",
       "  'life.',\n",
       "  'It',\n",
       "  'is',\n",
       "  'a',\n",
       "  'means',\n",
       "  'of',\n",
       "  'seeing',\n",
       "  'God',\n",
       "  'in',\n",
       "  'all',\n",
       "  'things',\n",
       "  'and',\n",
       "  'in',\n",
       "  'everyone.',\n",
       "  'Contemplation',\n",
       "  'is',\n",
       "  'a',\n",
       "  'critical',\n",
       "  'dimension',\n",
       "  'of',\n",
       "  'the',\n",
       "  'spiritual',\n",
       "  'life',\n",
       "  'and',\n",
       "  'it',\n",
       "  'is',\n",
       "  'reflected',\n",
       "  'in',\n",
       "  \"USF's\",\n",
       "  'commitment',\n",
       "  'to',\n",
       "  'prayer',\n",
       "  'and',\n",
       "  'spiritual',\n",
       "  'growth.',\n",
       "  'Analogously,',\n",
       "  'in',\n",
       "  'the',\n",
       "  'academic',\n",
       "  'life,',\n",
       "  'a',\n",
       "  'spirit',\n",
       "  'of',\n",
       "  'reflectivity',\n",
       "  'is',\n",
       "  'a',\n",
       "  'critical',\n",
       "  'aspect',\n",
       "  'of',\n",
       "  'intellectual',\n",
       "  'inquiry.\",',\n",
       "  \"u'',\",\n",
       "  \"u'2.\",\n",
       "  'Academic',\n",
       "  'Excellence',\n",
       "  '-',\n",
       "  'In',\n",
       "  '1547,',\n",
       "  'the',\n",
       "  'first',\n",
       "  'Jesuits',\n",
       "  'were',\n",
       "  'invited',\n",
       "  'to',\n",
       "  'begin',\n",
       "  'a',\n",
       "  'college',\n",
       "  'in',\n",
       "  'Messina,',\n",
       "  'Italy,',\n",
       "  'so',\n",
       "  'that',\n",
       "  'the',\n",
       "  'young',\n",
       "  'men',\n",
       "  'of',\n",
       "  'that',\n",
       "  'town',\n",
       "  'could',\n",
       "  'receive',\n",
       "  'the',\n",
       "  'same',\n",
       "  'quality',\n",
       "  'of',\n",
       "  'education',\n",
       "  'that',\n",
       "  'the',\n",
       "  'early',\n",
       "  'Jesuits',\n",
       "  'promoted',\n",
       "  'in',\n",
       "  'training',\n",
       "  'their',\n",
       "  'own',\n",
       "  'men.',\n",
       "  'USF',\n",
       "  'is',\n",
       "  'a',\n",
       "  'descendant',\n",
       "  'of',\n",
       "  'this',\n",
       "  'original',\n",
       "  'Jesuit',\n",
       "  'commitment',\n",
       "  'to',\n",
       "  'education.',\n",
       "  'Academic',\n",
       "  'excellence',\n",
       "  'describes',\n",
       "  'the',\n",
       "  'great',\n",
       "  'importance',\n",
       "  'that',\n",
       "  'Jesuits',\n",
       "  'have',\n",
       "  'placed',\n",
       "  'on',\n",
       "  'the',\n",
       "  'life',\n",
       "  'of',\n",
       "  'the',\n",
       "  'mind',\n",
       "  'as',\n",
       "  'a',\n",
       "  'means',\n",
       "  'for',\n",
       "  'uncovering',\n",
       "  'truth',\n",
       "  'and',\n",
       "  'discovering',\n",
       "  'meaning.',\n",
       "  'It',\n",
       "  'is',\n",
       "  'this',\n",
       "  'tradition',\n",
       "  'that',\n",
       "  'imbues',\n",
       "  'the',\n",
       "  'other',\n",
       "  '27',\n",
       "  'Jesuit',\n",
       "  'universities',\n",
       "  'and',\n",
       "  'colleges,',\n",
       "  'as',\n",
       "  'well',\n",
       "  'as',\n",
       "  'the',\n",
       "  'Jesuit',\n",
       "  'high',\n",
       "  'schools',\n",
       "  'and',\n",
       "  'middle',\n",
       "  'schools',\n",
       "  'in',\n",
       "  'the',\n",
       "  \"US.',\",\n",
       "  \"u'',\",\n",
       "  'u\"3.',\n",
       "  'Educating',\n",
       "  'the',\n",
       "  'Whole',\n",
       "  'Person',\n",
       "  '-',\n",
       "  'St.',\n",
       "  'Ignatius',\n",
       "  'believed',\n",
       "  'that',\n",
       "  'God',\n",
       "  'could',\n",
       "  'be',\n",
       "  'discovered',\n",
       "  'in',\n",
       "  'every',\n",
       "  'human',\n",
       "  'endeavor,',\n",
       "  'in',\n",
       "  'every',\n",
       "  'facet',\n",
       "  'of',\n",
       "  'learning',\n",
       "  'and',\n",
       "  'experience,',\n",
       "  'and',\n",
       "  'in',\n",
       "  'every',\n",
       "  'field',\n",
       "  'of',\n",
       "  'study.',\n",
       "  'Consequently,',\n",
       "  'he',\n",
       "  'promoted',\n",
       "  'the',\n",
       "  'development',\n",
       "  'of',\n",
       "  'the',\n",
       "  'spiritual,',\n",
       "  'intellectual,',\n",
       "  'artistic,',\n",
       "  'social,',\n",
       "  'and',\n",
       "  'physical',\n",
       "  'aspects',\n",
       "  'of',\n",
       "  'each',\n",
       "  'person.',\n",
       "  'The',\n",
       "  'commitment',\n",
       "  'to',\n",
       "  'educating',\n",
       "  'the',\n",
       "  'whole',\n",
       "  'person',\n",
       "  'is',\n",
       "  'evident',\n",
       "  'in',\n",
       "  \"USF's\",\n",
       "  'strong',\n",
       "  'core',\n",
       "  'curriculum',\n",
       "  'and',\n",
       "  'its',\n",
       "  'wide',\n",
       "  'array',\n",
       "  'of',\n",
       "  'learning',\n",
       "  'activities',\n",
       "  'and',\n",
       "  'experiences',\n",
       "  'that',\n",
       "  'bridges',\n",
       "  'cultures',\n",
       "  'and',\n",
       "  'national',\n",
       "  'boundaries.\",',\n",
       "  \"u'',\",\n",
       "  \"u'4.\",\n",
       "  '\"Cura',\n",
       "  'Personalis\"',\n",
       "  '-',\n",
       "  '\"Care',\n",
       "  'of',\n",
       "  'the',\n",
       "  'Person\"',\n",
       "  'was',\n",
       "  'originally',\n",
       "  'used',\n",
       "  'to',\n",
       "  'describe',\n",
       "  'the',\n",
       "  'responsibility',\n",
       "  'of',\n",
       "  'the',\n",
       "  'Jesuit',\n",
       "  'Superior',\n",
       "  'to',\n",
       "  'care',\n",
       "  'for',\n",
       "  'each',\n",
       "  'man',\n",
       "  'in',\n",
       "  'the',\n",
       "  'community',\n",
       "  'with',\n",
       "  'his',\n",
       "  'unique',\n",
       "  'gifts,',\n",
       "  'challenges,',\n",
       "  'needs,',\n",
       "  'and',\n",
       "  'possibilities.',\n",
       "  'This',\n",
       "  'value',\n",
       "  'now',\n",
       "  'is',\n",
       "  'applied',\n",
       "  'more',\n",
       "  'broadly',\n",
       "  'to',\n",
       "  'include',\n",
       "  'the',\n",
       "  'relationship',\n",
       "  'between',\n",
       "  'educators',\n",
       "  'and',\n",
       "  'students,',\n",
       "  'as',\n",
       "  'well',\n",
       "  'as',\n",
       "  'the',\n",
       "  'professional',\n",
       "  'relationships',\n",
       "  'among',\n",
       "  'all',\n",
       "  'those',\n",
       "  'who',\n",
       "  'work',\n",
       "  'in',\n",
       "  'the',\n",
       "  'University.',\n",
       "  '\"Cura',\n",
       "  'Personalis\"',\n",
       "  'suggests',\n",
       "  'individualized',\n",
       "  'attention',\n",
       "  'to',\n",
       "  'the',\n",
       "  'needs',\n",
       "  'of',\n",
       "  'the',\n",
       "  'other,',\n",
       "  'distinct',\n",
       "  'respect',\n",
       "  'for',\n",
       "  'his',\n",
       "  'or',\n",
       "  'her',\n",
       "  'unique',\n",
       "  'circumstances',\n",
       "  'and',\n",
       "  'concerns,',\n",
       "  'and',\n",
       "  'an',\n",
       "  'appropriate',\n",
       "  'appreciation',\n",
       "  'for',\n",
       "  'his',\n",
       "  'or',\n",
       "  'her',\n",
       "  'particular',\n",
       "  'gifts',\n",
       "  'and',\n",
       "  \"insights.']\"],\n",
       " [\"[u'',\",\n",
       "  \"u'5.\",\n",
       "  'Women',\n",
       "  'and',\n",
       "  'Men',\n",
       "  'for',\n",
       "  'Others',\n",
       "  '-',\n",
       "  'Fr.',\n",
       "  'Pedro',\n",
       "  'Arrupe,',\n",
       "  'S.J.,',\n",
       "  'Superior',\n",
       "  'General',\n",
       "  'of',\n",
       "  'the',\n",
       "  'Society',\n",
       "  'of',\n",
       "  'Jesuis',\n",
       "  'from',\n",
       "  '1965',\n",
       "  'to',\n",
       "  '1981,',\n",
       "  'employed',\n",
       "  'the',\n",
       "  'phrase',\n",
       "  '\"Men',\n",
       "  'for',\n",
       "  'Others\"',\n",
       "  'in',\n",
       "  'a',\n",
       "  'notable',\n",
       "  '1973',\n",
       "  'presentation',\n",
       "  'in',\n",
       "  'Valencia,',\n",
       "  'Spain.',\n",
       "  'Father',\n",
       "  'Arrupe',\n",
       "  'provocatively',\n",
       "  'challenged',\n",
       "  'the',\n",
       "  'alumni',\n",
       "  'of',\n",
       "  'Jesuit',\n",
       "  'schools',\n",
       "  'and',\n",
       "  'universities',\n",
       "  'to',\n",
       "  'be',\n",
       "  'engaged',\n",
       "  'in',\n",
       "  'the',\n",
       "  'struggle',\n",
       "  'for',\n",
       "  'justice',\n",
       "  'to',\n",
       "  'protect',\n",
       "  'the',\n",
       "  'needs',\n",
       "  'of',\n",
       "  'those',\n",
       "  'people',\n",
       "  'who',\n",
       "  'are',\n",
       "  'most',\n",
       "  'vulnerable.',\n",
       "  'Today',\n",
       "  'this',\n",
       "  'phrase',\n",
       "  'has',\n",
       "  'become',\n",
       "  'more',\n",
       "  'inclusive',\n",
       "  'and',\n",
       "  'its',\n",
       "  'spirit',\n",
       "  'is',\n",
       "  'evidenced',\n",
       "  'in',\n",
       "  \"USF\\\\'s\",\n",
       "  'promotion',\n",
       "  'of',\n",
       "  'service',\n",
       "  'projects;',\n",
       "  'justice',\n",
       "  'learning',\n",
       "  'through',\n",
       "  'immersion;',\n",
       "  'and',\n",
       "  'its',\n",
       "  'challenge',\n",
       "  'to',\n",
       "  'students,',\n",
       "  'faculty,',\n",
       "  'staff,',\n",
       "  'and',\n",
       "  'alumni',\n",
       "  'to',\n",
       "  '\"Change',\n",
       "  'the',\n",
       "  'World',\n",
       "  'from',\n",
       "  'Here.\"\\',',\n",
       "  \"u'',\",\n",
       "  \"u'6.\",\n",
       "  'Interreligious',\n",
       "  'Understanding',\n",
       "  '-',\n",
       "  'Reflecting',\n",
       "  'themes',\n",
       "  'from',\n",
       "  'the',\n",
       "  'Second',\n",
       "  'Vatican',\n",
       "  'Council,',\n",
       "  'the',\n",
       "  '34th',\n",
       "  'General',\n",
       "  'Congregation',\n",
       "  'of',\n",
       "  'the',\n",
       "  'Society',\n",
       "  'of',\n",
       "  'Jesus',\n",
       "  'made',\n",
       "  'a',\n",
       "  'significant',\n",
       "  'commitment',\n",
       "  'to',\n",
       "  'ecumenical',\n",
       "  'and',\n",
       "  'interreligious',\n",
       "  'engagement',\n",
       "  'and',\n",
       "  'understanding.',\n",
       "  'The',\n",
       "  'USF',\n",
       "  'community',\n",
       "  'comprises',\n",
       "  'a',\n",
       "  'wide',\n",
       "  'variety',\n",
       "  'of',\n",
       "  'religious',\n",
       "  'traditions;',\n",
       "  'and',\n",
       "  'through',\n",
       "  'its',\n",
       "  'University',\n",
       "  'Ministry,',\n",
       "  'it',\n",
       "  'encourages',\n",
       "  'people',\n",
       "  'of',\n",
       "  'different',\n",
       "  'faiths-and',\n",
       "  'no',\n",
       "  'faith-to',\n",
       "  'create',\n",
       "  'an',\n",
       "  'inclusive',\n",
       "  'community',\n",
       "  'that',\n",
       "  'is',\n",
       "  'reflective',\n",
       "  'and',\n",
       "  'authentic',\n",
       "  'in',\n",
       "  'its',\n",
       "  'search',\n",
       "  'for',\n",
       "  'meaning',\n",
       "  'and',\n",
       "  \"truth.',\",\n",
       "  \"u'',\",\n",
       "  'u\"7.',\n",
       "  'Community',\n",
       "  'in',\n",
       "  'Diversity',\n",
       "  '-',\n",
       "  'As',\n",
       "  'a',\n",
       "  'Catholic',\n",
       "  'and',\n",
       "  'Jesuit',\n",
       "  'University,',\n",
       "  'USF',\n",
       "  'welcomes',\n",
       "  'and',\n",
       "  'sustains',\n",
       "  'the',\n",
       "  'rich',\n",
       "  'diversity',\n",
       "  'among',\n",
       "  'the',\n",
       "  'students,',\n",
       "  'faculty,',\n",
       "  'and',\n",
       "  'staff',\n",
       "  'that',\n",
       "  'comprises',\n",
       "  'its',\n",
       "  'community.',\n",
       "  'The',\n",
       "  'USF',\n",
       "  'community',\n",
       "  'continues',\n",
       "  'to',\n",
       "  'be',\n",
       "  'enriched',\n",
       "  'by',\n",
       "  'people',\n",
       "  'of',\n",
       "  'different',\n",
       "  'abilities,',\n",
       "  'ages,',\n",
       "  'colors,',\n",
       "  'creeds,',\n",
       "  'faiths,',\n",
       "  'philosophies,',\n",
       "  'gender',\n",
       "  'orientations,',\n",
       "  'cultures,',\n",
       "  'nationalities,',\n",
       "  'family',\n",
       "  'backgrounds,',\n",
       "  'and',\n",
       "  'educational',\n",
       "  'experiences.',\n",
       "  'U.S.',\n",
       "  'News',\n",
       "  'and',\n",
       "  'World',\n",
       "  'Report',\n",
       "  'ranks',\n",
       "  'USF',\n",
       "  'among',\n",
       "  'the',\n",
       "  \"nation's\",\n",
       "  'top',\n",
       "  '30',\n",
       "  'university',\n",
       "  'for',\n",
       "  'student',\n",
       "  'ethnic',\n",
       "  'diversity',\n",
       "  'and',\n",
       "  'international',\n",
       "  'student',\n",
       "  'enrollment.\",',\n",
       "  \"u'***********************************************************************************']\"]]"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example 1-5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def split_function_index(index, partition):\n",
    "        word =  str(list(partition)).split()\n",
    "        output =  str(index) + \":\" + str(word) \n",
    "        yield output "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "words = lines.mapPartitionsWithIndex(split_function_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0:[\"[u\\'=\", \\'Ignatian\\', \\'Values\\', \"=\\',\", \"u\\'The\", \\'University\\', \\'of\\', \\'San\\', \\'Francisco\\', \\'enjoys\\', \\'a\\', \\'distinguished\\', \\'heritage\\', \\'and\\', \\'Jesuit\\', \\'tradition.\\', \\'At\\', \\'the\\', \\'core\\', \\'of\\', \\'this\\', \\'tradition\\', \\'are\\', \\'transcendent\\', \\'values,\\', \\'including\\', \\'the\\', \\'integration\\', \\'of\\', \\'learning,\\', \\'faith\\', \\'and\\', \\'service;\\', \\'care\\', \\'for\\', \\'the\\', \\'whole\\', \\'person;\\', \\'character\\', \\'and\\', \\'conviction;\\', \\'religious\\', \\'truth\\', \\'and\\', \\'interfaith\\', \\'understanding;\\', \\'and\\', \\'a\\', \\'commitment\\', \\'to\\', \\'building\\', \\'a\\', \\'more\\', \\'just\\', \\'world.\\', \\'The\\', \\'key\\', \\'values\\', \\'of\\', \\'this\\', \\'Jesuit\\', \\'tradition\\', \\'are\\', \\'as\\', \"follows:\\',\", \"u\\'***********************************************************************************\\',\", \\'u\"1.\\', \\'Contemplative\\', \\'in\\', \\'Action\\', \\'-\\', \\'St.\\', \\'Ignatius\\', \\'Loyola\\', \\'believed\\', \\'that\\', \\'prayer\\', \\'and\\', \\'reflectivity\\', \\'should\\', \\'so\\', \\'guide\\', \\'our\\', \\'choices\\', \\'and\\', \\'actions\\', \\'that\\', \\'our\\', \\'activity\\', \\'itself\\', \\'becomes\\', \\'a\\', \\'way\\', \\'of\\', \\'entering\\', \\'into\\', \\'union\\', \\'with\\', \\'and\\', \\'praising\\', \\'God.\\', \\'Being\\', \\'a\\', \\'contemplative\\', \\'in\\', \\'action\\', \\'also\\', \\'means\\', \\'seeing\\', \\'beyond\\', \\'the\\', \\'superficial\\', \\'in\\', \\'life\\', \\'to\\', \\'appreciate\\', \\'the\\', \\'mystery,\\', \\'beauty,\\', \\'and\\', \\'sacredness\\', \\'of\\', \\'all\\', \\'life.\\', \\'It\\', \\'is\\', \\'a\\', \\'means\\', \\'of\\', \\'seeing\\', \\'God\\', \\'in\\', \\'all\\', \\'things\\', \\'and\\', \\'in\\', \\'everyone.\\', \\'Contemplation\\', \\'is\\', \\'a\\', \\'critical\\', \\'dimension\\', \\'of\\', \\'the\\', \\'spiritual\\', \\'life\\', \\'and\\', \\'it\\', \\'is\\', \\'reflected\\', \\'in\\', \"USF\\'s\", \\'commitment\\', \\'to\\', \\'prayer\\', \\'and\\', \\'spiritual\\', \\'growth.\\', \\'Analogously,\\', \\'in\\', \\'the\\', \\'academic\\', \\'life,\\', \\'a\\', \\'spirit\\', \\'of\\', \\'reflectivity\\', \\'is\\', \\'a\\', \\'critical\\', \\'aspect\\', \\'of\\', \\'intellectual\\', \\'inquiry.\",\\', \"u\\'\\',\", \"u\\'2.\", \\'Academic\\', \\'Excellence\\', \\'-\\', \\'In\\', \\'1547,\\', \\'the\\', \\'first\\', \\'Jesuits\\', \\'were\\', \\'invited\\', \\'to\\', \\'begin\\', \\'a\\', \\'college\\', \\'in\\', \\'Messina,\\', \\'Italy,\\', \\'so\\', \\'that\\', \\'the\\', \\'young\\', \\'men\\', \\'of\\', \\'that\\', \\'town\\', \\'could\\', \\'receive\\', \\'the\\', \\'same\\', \\'quality\\', \\'of\\', \\'education\\', \\'that\\', \\'the\\', \\'early\\', \\'Jesuits\\', \\'promoted\\', \\'in\\', \\'training\\', \\'their\\', \\'own\\', \\'men.\\', \\'USF\\', \\'is\\', \\'a\\', \\'descendant\\', \\'of\\', \\'this\\', \\'original\\', \\'Jesuit\\', \\'commitment\\', \\'to\\', \\'education.\\', \\'Academic\\', \\'excellence\\', \\'describes\\', \\'the\\', \\'great\\', \\'importance\\', \\'that\\', \\'Jesuits\\', \\'have\\', \\'placed\\', \\'on\\', \\'the\\', \\'life\\', \\'of\\', \\'the\\', \\'mind\\', \\'as\\', \\'a\\', \\'means\\', \\'for\\', \\'uncovering\\', \\'truth\\', \\'and\\', \\'discovering\\', \\'meaning.\\', \\'It\\', \\'is\\', \\'this\\', \\'tradition\\', \\'that\\', \\'imbues\\', \\'the\\', \\'other\\', \\'27\\', \\'Jesuit\\', \\'universities\\', \\'and\\', \\'colleges,\\', \\'as\\', \\'well\\', \\'as\\', \\'the\\', \\'Jesuit\\', \\'high\\', \\'schools\\', \\'and\\', \\'middle\\', \\'schools\\', \\'in\\', \\'the\\', \"US.\\',\", \"u\\'\\',\", \\'u\"3.\\', \\'Educating\\', \\'the\\', \\'Whole\\', \\'Person\\', \\'-\\', \\'St.\\', \\'Ignatius\\', \\'believed\\', \\'that\\', \\'God\\', \\'could\\', \\'be\\', \\'discovered\\', \\'in\\', \\'every\\', \\'human\\', \\'endeavor,\\', \\'in\\', \\'every\\', \\'facet\\', \\'of\\', \\'learning\\', \\'and\\', \\'experience,\\', \\'and\\', \\'in\\', \\'every\\', \\'field\\', \\'of\\', \\'study.\\', \\'Consequently,\\', \\'he\\', \\'promoted\\', \\'the\\', \\'development\\', \\'of\\', \\'the\\', \\'spiritual,\\', \\'intellectual,\\', \\'artistic,\\', \\'social,\\', \\'and\\', \\'physical\\', \\'aspects\\', \\'of\\', \\'each\\', \\'person.\\', \\'The\\', \\'commitment\\', \\'to\\', \\'educating\\', \\'the\\', \\'whole\\', \\'person\\', \\'is\\', \\'evident\\', \\'in\\', \"USF\\'s\", \\'strong\\', \\'core\\', \\'curriculum\\', \\'and\\', \\'its\\', \\'wide\\', \\'array\\', \\'of\\', \\'learning\\', \\'activities\\', \\'and\\', \\'experiences\\', \\'that\\', \\'bridges\\', \\'cultures\\', \\'and\\', \\'national\\', \\'boundaries.\",\\', \"u\\'\\',\", \"u\\'4.\", \\'\"Cura\\', \\'Personalis\"\\', \\'-\\', \\'\"Care\\', \\'of\\', \\'the\\', \\'Person\"\\', \\'was\\', \\'originally\\', \\'used\\', \\'to\\', \\'describe\\', \\'the\\', \\'responsibility\\', \\'of\\', \\'the\\', \\'Jesuit\\', \\'Superior\\', \\'to\\', \\'care\\', \\'for\\', \\'each\\', \\'man\\', \\'in\\', \\'the\\', \\'community\\', \\'with\\', \\'his\\', \\'unique\\', \\'gifts,\\', \\'challenges,\\', \\'needs,\\', \\'and\\', \\'possibilities.\\', \\'This\\', \\'value\\', \\'now\\', \\'is\\', \\'applied\\', \\'more\\', \\'broadly\\', \\'to\\', \\'include\\', \\'the\\', \\'relationship\\', \\'between\\', \\'educators\\', \\'and\\', \\'students,\\', \\'as\\', \\'well\\', \\'as\\', \\'the\\', \\'professional\\', \\'relationships\\', \\'among\\', \\'all\\', \\'those\\', \\'who\\', \\'work\\', \\'in\\', \\'the\\', \\'University.\\', \\'\"Cura\\', \\'Personalis\"\\', \\'suggests\\', \\'individualized\\', \\'attention\\', \\'to\\', \\'the\\', \\'needs\\', \\'of\\', \\'the\\', \\'other,\\', \\'distinct\\', \\'respect\\', \\'for\\', \\'his\\', \\'or\\', \\'her\\', \\'unique\\', \\'circumstances\\', \\'and\\', \\'concerns,\\', \\'and\\', \\'an\\', \\'appropriate\\', \\'appreciation\\', \\'for\\', \\'his\\', \\'or\\', \\'her\\', \\'particular\\', \\'gifts\\', \\'and\\', \"insights.\\']\"]',\n",
       " '1:[\"[u\\'\\',\", \"u\\'5.\", \\'Women\\', \\'and\\', \\'Men\\', \\'for\\', \\'Others\\', \\'-\\', \\'Fr.\\', \\'Pedro\\', \\'Arrupe,\\', \\'S.J.,\\', \\'Superior\\', \\'General\\', \\'of\\', \\'the\\', \\'Society\\', \\'of\\', \\'Jesuis\\', \\'from\\', \\'1965\\', \\'to\\', \\'1981,\\', \\'employed\\', \\'the\\', \\'phrase\\', \\'\"Men\\', \\'for\\', \\'Others\"\\', \\'in\\', \\'a\\', \\'notable\\', \\'1973\\', \\'presentation\\', \\'in\\', \\'Valencia,\\', \\'Spain.\\', \\'Father\\', \\'Arrupe\\', \\'provocatively\\', \\'challenged\\', \\'the\\', \\'alumni\\', \\'of\\', \\'Jesuit\\', \\'schools\\', \\'and\\', \\'universities\\', \\'to\\', \\'be\\', \\'engaged\\', \\'in\\', \\'the\\', \\'struggle\\', \\'for\\', \\'justice\\', \\'to\\', \\'protect\\', \\'the\\', \\'needs\\', \\'of\\', \\'those\\', \\'people\\', \\'who\\', \\'are\\', \\'most\\', \\'vulnerable.\\', \\'Today\\', \\'this\\', \\'phrase\\', \\'has\\', \\'become\\', \\'more\\', \\'inclusive\\', \\'and\\', \\'its\\', \\'spirit\\', \\'is\\', \\'evidenced\\', \\'in\\', \"USF\\\\\\\\\\'s\", \\'promotion\\', \\'of\\', \\'service\\', \\'projects;\\', \\'justice\\', \\'learning\\', \\'through\\', \\'immersion;\\', \\'and\\', \\'its\\', \\'challenge\\', \\'to\\', \\'students,\\', \\'faculty,\\', \\'staff,\\', \\'and\\', \\'alumni\\', \\'to\\', \\'\"Change\\', \\'the\\', \\'World\\', \\'from\\', \\'Here.\"\\\\\\',\\', \"u\\'\\',\", \"u\\'6.\", \\'Interreligious\\', \\'Understanding\\', \\'-\\', \\'Reflecting\\', \\'themes\\', \\'from\\', \\'the\\', \\'Second\\', \\'Vatican\\', \\'Council,\\', \\'the\\', \\'34th\\', \\'General\\', \\'Congregation\\', \\'of\\', \\'the\\', \\'Society\\', \\'of\\', \\'Jesus\\', \\'made\\', \\'a\\', \\'significant\\', \\'commitment\\', \\'to\\', \\'ecumenical\\', \\'and\\', \\'interreligious\\', \\'engagement\\', \\'and\\', \\'understanding.\\', \\'The\\', \\'USF\\', \\'community\\', \\'comprises\\', \\'a\\', \\'wide\\', \\'variety\\', \\'of\\', \\'religious\\', \\'traditions;\\', \\'and\\', \\'through\\', \\'its\\', \\'University\\', \\'Ministry,\\', \\'it\\', \\'encourages\\', \\'people\\', \\'of\\', \\'different\\', \\'faiths-and\\', \\'no\\', \\'faith-to\\', \\'create\\', \\'an\\', \\'inclusive\\', \\'community\\', \\'that\\', \\'is\\', \\'reflective\\', \\'and\\', \\'authentic\\', \\'in\\', \\'its\\', \\'search\\', \\'for\\', \\'meaning\\', \\'and\\', \"truth.\\',\", \"u\\'\\',\", \\'u\"7.\\', \\'Community\\', \\'in\\', \\'Diversity\\', \\'-\\', \\'As\\', \\'a\\', \\'Catholic\\', \\'and\\', \\'Jesuit\\', \\'University,\\', \\'USF\\', \\'welcomes\\', \\'and\\', \\'sustains\\', \\'the\\', \\'rich\\', \\'diversity\\', \\'among\\', \\'the\\', \\'students,\\', \\'faculty,\\', \\'and\\', \\'staff\\', \\'that\\', \\'comprises\\', \\'its\\', \\'community.\\', \\'The\\', \\'USF\\', \\'community\\', \\'continues\\', \\'to\\', \\'be\\', \\'enriched\\', \\'by\\', \\'people\\', \\'of\\', \\'different\\', \\'abilities,\\', \\'ages,\\', \\'colors,\\', \\'creeds,\\', \\'faiths,\\', \\'philosophies,\\', \\'gender\\', \\'orientations,\\', \\'cultures,\\', \\'nationalities,\\', \\'family\\', \\'backgrounds,\\', \\'and\\', \\'educational\\', \\'experiences.\\', \\'U.S.\\', \\'News\\', \\'and\\', \\'World\\', \\'Report\\', \\'ranks\\', \\'USF\\', \\'among\\', \\'the\\', \"nation\\'s\", \\'top\\', \\'30\\', \\'university\\', \\'for\\', \\'student\\', \\'ethnic\\', \\'diversity\\', \\'and\\', \\'international\\', \\'student\\', \\'enrollment.\",\\', \"u\\'***********************************************************************************\\']\"]']"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example 1-6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "words = lines.flatMap(lambda line : line.split(\" \"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'learning,', u'of', u'comprises']"
      ]
     },
     "execution_count": 203,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words.sample(True,0.01,1).collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'learning,', u'of', u'comprises']"
      ]
     },
     "execution_count": 204,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words.sample(True,0.01,1).collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'=',\n",
       " u'should',\n",
       " u'a',\n",
       " u'Person\"',\n",
       " u'circumstances',\n",
       " u'a',\n",
       " u'and',\n",
       " u'by',\n",
       " u'***********************************************************************************']"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words.sample(False,0.01,1).collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'=',\n",
       " u'should',\n",
       " u'a',\n",
       " u'Person\"',\n",
       " u'circumstances',\n",
       " u'a',\n",
       " u'and',\n",
       " u'by',\n",
       " u'***********************************************************************************']"
      ]
     },
     "execution_count": 206,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words.sample(False,0.01,1).collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}