adding wordpress instructions, check for description in post page, test command to generate posts, and removal of ? in filenames

Signed-off-by: Vanessa Sochat <[email protected]>

Author: vsoch

README.md

@@ -20,7 +20,19 @@ of what we collect:
 ```
 
 The tag must be unique (and this is tested), and the feed should be a format
-parseable by [feedparser](https://pythonhosted.org/feedparser/) (most are). 
+parseable by [feedparser](https://pythonhosted.org/feedparser/) (most are).
+
+#### Wordpress
+
+WordPress is a common blogging platform, and so we include notes here for how
+to find a feed for your wordpress blog. If you want to include all content,
+you can usually find a main feed at `https://<yourblog>/feed/`. However, it's
+recommended to create a [tag or category](https://wordpress.org/support/article/wordpress-feeds/#categories-and-tags) 
+feed, in which case you could find the feed at `https://<yourblog>/category/<category>/feed/`. See the linked
+page for more ways that you can generate custom feeds based on tags and categories.
+Once you've added your feed, it's recommended to test generate posts to ensure
+that it's parsed correctly. This is done during the continuous integration,
+but you can also do it locally (see below).
 
 ### 2. Generate Posts
 
@@ -40,7 +52,26 @@ For this reason, we also define this in the context.
 
 The reason this is set up to run with continuous integration is so that the site
 is regularly updated without human intervention. In the case that there is an error,
-the maintainers are notified.
+the maintainers are notified. However, you can test this generation locally, without
+generating any files!
+
+```bash
+$ python script/generate_posts.py _data/authors.yml --output _posts/ --test
+```
+
+It will show you any new folders and files generated without actually doing it.
+
+```bash
+$ python script/generate_posts.py _data/authors.yml --output _posts/ --test
+[TEST] new author folder /home/vanessa/Documents/Dropbox/Code/usrse/blog/_posts/dsk
+Preparing new post: /home/vanessa/Documents/Dropbox/Code/usrse/blog/_posts/dsk/2019-4-22-p=1452.md
+Preparing new post: /home/vanessa/Documents/Dropbox/Code/usrse/blog/_posts/dsk/2019-2-5-p=1446.md
+Preparing new post: /home/vanessa/Documents/Dropbox/Code/usrse/blog/_posts/dsk/2019-1-23-p=1444.md
+Preparing new post: /home/vanessa/Documents/Dropbox/Code/usrse/blog/_posts/dsk/2018-9-26-p=1442.md
+Preparing new post: /home/vanessa/Documents/Dropbox/Code/usrse/blog/_posts/dsk/2018-6-27-p=1423.md
+Preparing new post: /home/vanessa/Documents/Dropbox/Code/usrse/blog/_posts/dsk/2018-6-25-p=1421.md
+Preparing new post: /home/vanessa/Documents/Dropbox/Code/usrse/blog/_posts/dsk/2018-2-8-p=1415.md
+```
 
 ## Development
 

_layouts/post.html

@@ -31,7 +31,7 @@ <h2 class="subheading">{{ page.subtitle }}</h2>
       <div class="col-lg-8 col-md-10 mx-auto">
 
         <p class="alert alert-primary">
-        This is a crosspost from <a href="{{ page.blog_url }}">{{ page.blog_title }}</a>, {{ page.blog_subtitle }}. 
+        This is a crosspost from <a href="{{ page.blog_url }}">{{ page.blog_title }}</a>{% if page.blog_subtitle %}, {{ page.blog_subtitle }}{% endif %}. 
         See the original post <a href="{{ page.original_url }}">here</a>.</p>
         </a>
 

script/generate_posts.py

@@ -30,6 +30,10 @@ def get_parser():
                         default=None, dest="output",
                         help="The output folder to write posts.")
 
+    parser.add_argument("--test", 
+                        default=False, dest="test",
+                        action='store_true',
+                        help="Test generation only (show files to be created)")
     return parser
 
 
@@ -61,14 +65,15 @@ def validate_authors(authors):
     return valid
 
 
-def parse_feeds(authors, output_dir):
+def parse_feeds(authors, output_dir, test=False):
     '''read in the list of authors, parse feeds and save results to a
        specified output directory.
  
        Parameters
        ==========
        authors: a list of authors, read in from an authors.yml
        output_dir: the output directory to write markdown posts to.
+       test: don't write any files, only test generate
     '''
     if output_dir == None:
         print("Output directory must be defined.")
@@ -84,8 +89,11 @@ def parse_feeds(authors, output_dir):
         # Create output folder
         author_folder = os.path.join(output_dir, author['tag'])
         if not os.path.exists(author_folder):
-            print("Creating new author folder %s" % author_folder)
-            os.mkdir(author_folder)
+            if test is False:
+                print("Creating new author folder %s" % author_folder)
+                os.mkdir(author_folder)
+            else:
+                print("[TEST] new author folder %s" % author_folder)
 
         # Parse the feed, each entry is written to file based on title
         feed = feedparser.parse(author['feed'])
@@ -95,12 +103,15 @@ def parse_feeds(authors, output_dir):
 
             # Write the file if it doesn't exist
             if not os.path.exists(markdown):
+
                 print('Preparing new post: %s' % markdown)
                 post = generate_post(entry, author, feed)
 
-                # Write to file
-                with open(markdown, 'w') as filey: 
-                    filey.write(frontmatter.dumps(post))
+                if test is False:
+
+                    # Write to file
+                    with open(markdown, 'w') as filey: 
+                        filey.write(frontmatter.dumps(post))
 
 
 def generate_post(entry, author, feed):
@@ -145,6 +156,9 @@ def get_markdown_file(author_folder, entry):
 
     # The id is the last part of the url, lowercase
     title = [x for x in entry['id'].split('/') if x][-1].lower()
+
+    # Replace any variable names (? in wordpress) with -
+    title = title.replace('?', '')
     filename = '%s-%s-%s-%s.md' %(year, month, day, title)
 
     # The output markdown name is consistent
@@ -206,7 +220,7 @@ def help(return_code=0):
         print("Authors file %s is not valid." % authors)
 
     # Generate outputs based on authors
-    parse_feeds(authors, args.output)
+    parse_feeds(authors, args.output, args.test)
 
 
 if __name__ == '__main__':