Merge pull request #28 from DynamicGravitySystems/develop

Merge ansible updates from develop into master

LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright 2018 Zachery P. Brady
+Copyright 2016-2020 Zachery P. Brady
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
 

MANIFEST.in

@@ -1,3 +1,3 @@
 include README.md
 include tools/send.py
-include atgmlogger/install/*
+include atgmlogger/atgmlogger.json

README.md

@@ -1,5 +1,5 @@
-Dynamic Gravity Systems - Serial Data Recorder
-==============================================
+Dynamic Gravity Systems - Serial Data Recorder v0.6.0
+=====================================================
 
 1. Dependencies:
  	- Python v3.5 or 3.6 or later, and the following modules:
@@ -9,96 +9,29 @@ Dynamic Gravity Systems - Serial Data Recorder
 		- ntfs-3g
 		- exfat-fuse
 		- exfat-utils
-2. Preparing the Raspberry Pi:
-	1. Installing Python3.6 from source:
-		- Download Python3.6 source tarball from https://www.python.org 
-		- Install the required development libraries to build the source:
-			- make
-			- build-essential
-			- libssl-dev
-			- zliblg-dev
-			- libbz2-dev
-			- libreadline-dev
-			- libsqlite3-dev
-			- wget
-			- curl
-			- llvm
-			- libncurses5-dev
-			- libncursesw5-dev
-			- xz-utils
-			- tk-dev
-		```commandline
-		sudo apt-get install -y make build-essential libssl-dev zlib1g-dev   
-		sudo apt-get install -y libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm 
-		sudo apt-get install -y libncurses5-dev  libncursesw5-dev xz-utils tk-dev
-		```
-		- Extract, Configure and Build:
-		```commandline
-		tar -xzf python-3.6.2.tgz 
-		cd python-3.6.2
-		./configure
-		make
-		sudo make altinstall
-		```
-
-	2. Install Python3.6 from Debian SID (experimental) repository:
-	    - add the following line to /etc/apt/sources.list:
-	    
-	        ```commandline
-            deb http://http.us.debian.org/debian sid main
-            ```
 
-        - configure apt-pinning to prevent the system from using experimental repo for system-wide updates
-        
-            ```commandline
-            vim /etc/apt/preferences.d/pinning
-            
-            Package: *
-            Pin: release a=stable
-            Pin-Priority: 700
-            ```
-            
-	4. **Configure Raspberry Pi GPIO Console:**
-		- By default the Raspberry Pi GPIO console is enabled as a TTY terminal, this needs to be disabled to allow it 
-		to be used as a Serial Data input.
-		- Modify /boot/cmdline.txt removing the section similar to: 'console=serial0,115200'
-			
-			```commandline(bash)
-				# All Commands executed as Root (sudo)
-				sed -i -e s/console=serial0,115200//g /boot/cmdline.txt
-				echo 'enable_uart=1' >> /boot/config.txt
-				systemctl stop [email protected]
-				systemctl disable [email protected]
-			``` 
-
-3. Installation:
-    - The atgmlogger utility is now packaged as a Python Wheel and can be installed via pip:
-    ```commandline
-    sudo pip3 install atgmlogger-0.3.1-py3-none-any.whl
-    ```	
-    This method will create a commandline script in /usr/bin/atgmlogger, which can be invoked to run the logger.
-	
-
-
-Manual Installation:
---------------------
-
-Installation Directories (copy the following files to the specified destinations):
-  - 90-removable-usb.rules -> /etc/udev/rules.d/90-removable-usb.rules
-  - media-removable.mount -> /etc/systemd/system/media-removable.mount
-  - SerialLogger.service -> /etc/systemd/system/SerialLogger.service
+2. Installation:
+
+    - The installation of atgmlogger on a Raspberry Pi requires several supporting configuration changes and updates; 
+    to automate the process an ansible playbook is provided: [ATGMLogger Ansible Playbook](https://github.com/DynamicGravitySystems/atgmlogger-ansible)
+
+    - In general the following steps are taken to prepare the system:
+        1. Configure values in /boot/cmdline.txt and /boot/config.txt to enable UART and disable the TTY on the serial GPIO
+        2. Install dependencies that enable the use of NTFS/FAT/EXFAT formatted external devices (for data retrieval)
+        3. Add a UDEV rule and systemd mount unit to auto-mount any removable block device (e.g. USB drive) in order to allow copying of logged data for retrieval
+        4. Add a logrotate configuration to automatically rotate data and application logs
+        5. Install python3 and the ATGMLogger python application
+        6. Install a systemd service unit file to control the automatic startup of the ATGMLogger application
+        7. Secure the raspberry Pi by changing the default 'pi' user password, and adding an authorized key for factory maintenance usage
+
 
-After installing .mount and .service files run the following commands:
-```commandline
-sudo systemctl daemon-reload
-sudo systemctl enable media-removable.mount
-sudo systemctl enable SerialLogger.service
-```
+3. Execution:
 
-Explanation:
-- 90-removable-usb.rules creates a UDEV rule that adds a symbolic link to /dev/usbstick when a usb block device (hdd)
-is inserted. This symlink is used by the following mount file to mount the filesystem.
-- media-removable.mount is a systemd mount unit which instructs systemd to mount /dev/usbstick to /media/removable when
-it detects the device 'dev-usbstick.device'. The unit will also dismount the device when it becomes unavailable.
-- SerialLogger.service is a systemd service unit which executes the Serial Logging python script. When this unit is
-enabled (see above command - systemctl enable SerialLogger.service) the program will be executed upon system startup.
+    - Once installed with the ansible deploy playbook, ATGMLogger will automatically start every time the system is booted
+    via a systemd service unit.
+
+    - ATGMLogger can otherwise be manually executed (e.g. for debugging purposes) with the following command
+
+        ```commandline
+        /usr/bin/python3 -m atgmlogger -vvv
+        ```

atgmlogger/__init__.py

@@ -3,13 +3,11 @@
 import sys
 import logging
 
-__all__ = ['LOG_LVLMAP', 'LOG_FMT', 'DATE_FMT', 'POSIX',
-           '__version__', '__description__']
+__all__ = ['LOG_LVLMAP', 'LOG_FMT', 'DATE_FMT', 'POSIX', '__version__', '__description__']
 
-__version__ = '0.4.5'
+__version__ = '0.6.0'
 __description__ = "Advanced Technology Gravity Meter - Serial Data Logger"
 
-
 LOG_LVLMAP = {0: logging.CRITICAL,
               1: logging.ERROR,
               2: logging.WARNING,
@@ -18,6 +16,8 @@
 # Application level root logger - all other loggers should branch from this
 APPLOG = logging.getLogger('atgmlogger')
 LOG_FMT = "%(levelname)8s::%(asctime)s - (%(funcName)s) %(message)s"
+TRACE_LOG_FMT = "%(levelname)8s::%(asctime)s - (%(name)s/%(funcName)s:%(lineno)d " \
+                "- thread: %(threadName)s) %(message)s"
 DATE_FMT = "%Y-%m-%d::%H:%M:%S"
 _stderr_hdlr = logging.StreamHandler(sys.stderr)
 _stderr_hdlr.setFormatter(logging.Formatter(LOG_FMT,

atgmlogger/__main__.py

@@ -1,9 +1,9 @@
 #! /usr/bin/python3
 # -*- encoding: utf-8 -*-
 
-import sys
-import logging
 import argparse
+import logging
+import sys
 from pathlib import Path
 
 from . import __description__, __version__, LOG_LVLMAP
@@ -28,41 +28,18 @@ def parse_args(argv=None):
     parser.add_argument('--trace', action='store_true',
                         help="Enable detailed trace info in log messages.")
 
-    # Sub-Parser Groups
-    sub_parsers = parser.add_subparsers(dest='command', help="Subcommands to run/install/uninstall ATGMLogger")
-
-    install_parser = sub_parsers.add_parser('install', help='Install system files for ATGMLogger', allow_abbrev=True)
-    install_parser.add_argument('--service', action='store_true', default=True, help='Install ATGMLogger as a Systemd '
-                                                                                     'Service')
-    install_parser.add_argument('--dependencies', action='store_true', help='Install system dependencies using apt.')
-    install_parser.add_argument('--configure', action='store_true', help='Run RaspberryPi configuration scripts.')
-    install_parser.add_argument('--check-install', action='store_true', help='Verify installed components.')
-    install_parser.add_argument('--logrotate', action='store_true', default=True, help='Install logrotate '
-                                                                                       'configuration')
-    install_parser.add_argument('--with-mqtt', action='store_true', help='Install MQTT plugin configuration and AWS '
-                                                                         'IoT dependencies')
-
-    uninst_parser = sub_parsers.add_parser('uninstall', help='Uninstall ATGMLogger system files and configurations')
-    uninst_parser.add_argument('--keep-config', action='store_true', help='Retain configuration after uninstalling '
-                                                                          'ATGMLogger')
-
-    chkinst_parser = sub_parsers.add_parser('chkinstall', help="Check ATGMLogger installation and depdendencies")
-
-    run_parser = sub_parsers.add_parser('run', help='Run atgmlogger')
-    run_parser.add_argument('-d', '--device', action='store',
+    # Runtime options
+    parser.add_argument('-d', '--device', action='store',
                             help="Serial device path")
-    run_parser.add_argument('-l', '--logdir', action='store')
-    run_parser.add_argument('-m', '--mountdir', action='store',
+    parser.add_argument('-l', '--logdir', action='store')
+    parser.add_argument('-m', '--mountdir', action='store',
                             help="Specify custom USB Storage mount path. "
                                  "Overrides path configured in configuration.")
-    run_parser.add_argument('-c', '--config', action='store',
+    parser.add_argument('-c', '--config', action='store',
                             help="Specify path to custom JSON configuration.")
-    run_parser.add_argument('--nogpio', action='store_true',
+    parser.add_argument('--nogpio', action='store_true',
                             help="Disable GPIO output (LED notifications).")
 
-    # This fails if we specify global positional args before the command
-    # if args[0].lower() not in ['install', 'uninstall', 'run']:
-    #     args.insert(0, 'run')
     return parser.parse_args(args)
 
 
@@ -75,37 +52,6 @@ def initialize(args):
         log_level = LOG_LVLMAP.get(args.verbose, logging.INFO)
     LOG.setLevel(log_level)
 
-    if args.command in {'install', 'uninstall', 'chkinstall'}:
-        from . import install
-        method = getattr(install, args.command, None)
-        if method is None:
-            LOG.error("Command %s is not implemented", args.command)
-            sys.exit(1)
-        sys.exit(method(args))
-
-
-    # if args.command == 'install':
-    #     try:
-    #         from .install import install
-    #         sys.exit(install(args))
-    #     except (ImportError, OSError):
-    #         LOG.exception("Exception occurred trying to install system "
-    #                       "files.")
-    #         sys.exit(1)
-    # elif args.command == 'uninstall':
-    #     try:
-    #         from .install import uninstall
-    #         sys.exit(uninstall(args))
-    #     except (ImportError, OSError):
-    #         LOG.exception("Exception occurred uninstalling system files.")
-    # elif args.command == 'chkinstall':
-    #     try:
-    #         from .install import chkinstall
-    #         sys.exit(chkinstall(args))
-    #
-    #     except (ImportError, OSError):
-    #         pass
-
     # Set overrides from arguments
     from .runconfig import rcParams
 
@@ -132,3 +78,7 @@ def entry_point():
     from .atgmlogger import atgmlogger
 
     sys.exit(atgmlogger(args))
+
+
+if __name__ == "__main__":
+    entry_point()

atgmlogger/atgmlogger.py

@@ -131,7 +131,6 @@ def decode(bytearr, encoding='utf-8'):
         return decoded
 
 
-# TODO: Move some/all of this functionality into __main__ initialize function
 def _configure_applog(log_format):
     logdir = Path(rcParams['logging.logdir'])
     if not logdir.exists():
@@ -207,11 +206,7 @@ def atgmlogger(args, listener=None, handle=None, dispatcher=None):
     # Init Performance Counter
     t_start = time.perf_counter()
 
-    # TODO: Again candidate to move into __main__::initialize
-    fmt = LOG_FMT
-    if args.trace:
-        fmt = TRACE_LOG_FMT
-    _configure_applog(fmt)
+    _configure_applog(TRACE_LOG_FMT if args.trace else LOG_FMT)
 
     if listener is None:
         listener = SerialListener(handle or _get_handle())
@@ -228,7 +223,6 @@ def atgmlogger(args, listener=None, handle=None, dispatcher=None):
             # Note: Signal handler must be defined in main thread
             signal.signal(signal.SIGHUP, lambda sig, frame: dispatcher.log_rotate())
         dispatcher.start()
-        # print(logging.Logger.manager.loggerDict.keys())
         listener()
     except KeyboardInterrupt:
         LOG.info("Keyboard Interrupt intercepted, cleaning up and exiting.")