You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
magedbm2 is a database backup manager for Magento 2.
5
+
Magedbm2 is a database backup manager designed to make the process of taking backups from one environment and moving them to another, easy. The backups can be stripped of sensitive tables and can even generate anonymised versions of the sensitive data. While it was originally developed to for Magento 2.x it can be used for any system.
8
6
9
7
It was written with developers in mind and provides commands for:
10
8
11
-
* Creating sanitised (or unsanitised) Magento 2 production database backups and uploading them to Amazon S3
9
+
* Creating sanitised (or unsanitised) production database backups and uploading them to Amazon S3
12
10
* Downloading and importing existing backups to development or staging environments
13
11
* Exporting and importing anonymised sensitive data tables
14
12
15
-
## Compatibility
13
+
## Requirements
16
14
17
-
magedbm2 requires PHP 7.0+ with permission to run the `exec` and `passthru` functions to create and import database backups.
15
+
* PHP 7.0+ (with permission to run `exec` and `passthru`)
16
+
*`mysql` client on the `$PATH`
18
17
19
18
## Installation
20
19
@@ -25,77 +24,56 @@ Download the Phar file from the [latest Github release](https://github.com/space
25
24
26
25
## Configuration
27
26
28
-
magedbm2 uses [Amazon S3](https://aws.amazon.com/s3/) to store generated files. To use it you will need to create two buckets on S3 (one for database backups and another for data exports) and a user that has read and write access to the buckets. Note down your:
29
-
30
-
- AWS Access Key ID
31
-
- AWS Secret Access Key
32
-
- S3 region where your bucket is hosted
33
-
- The bucket names
34
-
35
-
These details will need to be specified with option flags to most `magedbm2` commands.
36
-
37
-
### Remembering credentials
27
+
The configuration can either be provided through configuration files or as options when executing a command. For more information on the configuration files, see [Using Multiple Configuration Files](docs/multiple-configuration-files.md).
38
28
39
-
While the ability to specify the credentials as command line options is useful in automated scenarios, for personal everyday usage magedbm2 can be configured to remember your credentials by running:
29
+
The basic configuration parameters required are the credentials for connecting to [Amazon S3](https://aws.amazon.com/s3/) and which buckets to store the different types of exports in.
By default, the configure command will interactively prompt you for each of the configuration details. Alternatively, you can run it in a non-interactive mode with the `-n` option and specify the credentials you want to save by using the `--access-key`, `--secret-key`, `--region`, `--bucket` or `--data-bucket` options.
44
-
45
-
The configuration is saved in a [YAML](http://www.yaml.org/) file, usually located in `~/.magedbm2/config.yml`.
31
+
*`access-key`: Your AWS Access Key
32
+
*`secret-key`: Your AWS Secret Key
33
+
*`region`: The region in which the S3 buckets are located
34
+
*`bucket`: The bucket to store the database backups
35
+
*`data-bucket`: The bucket to store the anonymised exports
46
36
47
37
### Using without a Magento Installation
48
38
49
-
It's not always possible to run `magedbm2` on the same server that a Magento installation is present, for example you might want to run a cron for `magedbm2` on your database server so that you don't clog the pipe to the application server with unnecessary backup traffic.
39
+
It's not always possible to run Magedbm2 on the same server that a Magento installation is present, for example you might want to run a cron for Magedbm2 on your database server so that you don't clog the pipe to the application server with unnecessary export traffic.
50
40
51
-
`magedbm2` normally discovers the database credentials by looking for the `app/etc/env.php` configuration file in a Magento project. If this is not found then `magedbm2` will fallback to the following options:
41
+
Magedbm2 normally discovers the database credentials by looking for the `app/etc/env.php`(or `app/etc/local.xml` for Magento 1.x) configuration file in a Magento project. If this is not found then Magedbm2 will fallback to the following options:
52
42
53
43
*`--db-host`
54
44
*`--db-port`
55
45
*`--db-user`
56
46
*`--db-pass`
47
+
*`--db-name`
57
48
58
-
### Defining strip table rules
49
+
This gives us the ability to use Magedbm2 without Magento at all, should we want to.
59
50
60
-
Often in your projects you'll have a list of tables that you want to strip as a group. You may be familiar with this concept in [magerun](https://github.com/netz98/n98-magerun) when passing `--strip=@development` to the `db:dump` command. Magedbm ships with a default set of strip table groups but you can add your by adding a new key under `table-groups` in your configuration, e.g.
51
+
### Writing Configuration Files
61
52
62
-
table-groups:
63
-
- id: superpay
64
-
description: Sensitive Super Pay tables
65
-
tables: superpay_* super_logs superpay
53
+
While the ability to specify the credentials as command line options is useful in automated scenarios, for personal everyday usage Magedbm2 can be configured to remember your credentials by running:
66
54
67
-
You can then use that table group in your `put` command, `magedbm2 put --strip=@superpay myproject`.
By default, the configure command will interactively prompt you for each of the configuration details. Alternatively, you can run it in a non-interactive mode with the `-n` option and specify the credentials you want to save using the options.
70
58
71
-
As well as stripping certain tables in your projects, you might also have project-specific columns containing sensitive data that you do not want exported. To do this you need to define which formatter to apply to the column.
59
+
The configuration is saved in a [YAML](http://www.yaml.org/) file determined by the `file-name` argument. If running in interactive mode you will be prompted with appropriate file locations.
72
60
73
-
Depending on the type of entity that you want to anonymise you'll either need to use the `tables` or the `eav` key in the configuration. Assuming that we've added a new table called `super_pay` with two columns that need anonymisation and we've added a new attribute to customers.
There are some advanced configuration options that you may be interested in, such as:
85
64
86
-
You have access to the [Faker](https://github.com/fzaninotto/Faker) library.
87
-
88
-
The format of the formatter string is either `${className}` if the class implements `Meanbee\Magedbm2\Anonymizer\FormatterInterface` or `${className}::${method}` if you want to call a method directly.
65
+
*[Defining Table Groups for Stripping](docs/table-groups.md)
magedbm2 put [--strip="..."] [--clean=NUM|--no-clean] <project>
72
+
magedbm2 put [--strip="@development"] [--clean=NUM|--no-clean] <project>
95
73
96
-
The `put` command will create a database backup from a Magento 2 installation at the current working directory (or the directory specified with`--root-dir`) and upload it to S3.
74
+
The `put` command will create a database backup and upload it to S3. If no database parameters are passed through on the command line, Magento will be assumed, and the database details will attempted to be loaded from the current working directory, or the directory specified by`--root-dir`.
97
75
98
-
By default, the command uses [n98-magerun2](https://github.com/netz98/n98-magerun2) for database operations and creates backups with all customer data stripped out (the `@development` strip setting to the `n98-magerun2 db:dump` command). However, you can customise what data gets stripped, if any, using the `--strip` option.
76
+
By default, the tables defined in the `@development` table group (which can be seen by running `magedbm2 put --help`) will be stripped. Additional tables or table groups can be added to the `--strip` option, separated by spaces, e.g. `--strip="@log @sessions customtable customprefix*"`.
99
77
100
78
The `put` command also automatically cleans up old database backups once a new one has been uploaded, keeping only the 5 most recent backup files. You can customise the number of retained backups with the `--clean` option, or disable it completely with `--no-clean`.
101
79
@@ -109,7 +87,7 @@ The `ls` command will show the backup files available for import in S3. If calle
109
87
110
88
magedbm2 get [--download-only] [--force] get <project> [<file>]
111
89
112
-
Use the `get` command to download a backup file from S3 and import it to a Magento 2 installation at the current working directory (or the directory specified with`--root-dir`).
90
+
Use the `get` command to download a backup file from S3 and import it into the defined database. If no database parameters are passed through on the command line, Magento will be assumed, and the database details will attempted to be loaded from the current working directory, or the directory specified by`--root-dir`.
113
91
114
92
By default, the `get` command will choose the latest backup file available for the given project. You can import specific backup files by providing the name of the file after the project name.
115
93
@@ -133,62 +111,14 @@ The `export` command allows you generate an anonymised export of your database,
133
111
134
112
The `import` command allows you to import an anonymised export of your database.
135
113
136
-
## Development
137
-
138
-
### Contributing
139
-
140
-
While there is no formal contribution process, feel free to contribute by:
141
-
142
-
1. Creating issues, bug reports or feature requests on [Github](https://github.com/space48/magedbm2/issues)
143
-
2. Submitting pull requests for improvements
144
-
145
-
Pull requests should be submitted to the `develop` branch to be included in the next release.
146
-
147
-
### Requirements
148
-
149
-
Developing magedbm2 requires:
150
-
151
-
- PHP 7.0+
152
-
-[Phing](https://www.phing.info/)
153
-
154
-
### Installation
155
-
156
-
To install development dependencies for magedbm2, run:
114
+
### Viewing Configuration
157
115
158
-
phing install
159
-
160
-
### Code style
161
-
162
-
magedbm2 follows PSR-2 for code style and a set of rules from [phpmd](https://phpmd.org/) to check for code problems.
163
-
164
-
You should run the following to check for any potential issues with the code before committing it:
165
-
166
-
phing lint
167
-
168
-
### Testing
169
-
170
-
magedbm2 is tested with [PHPUnit](https://phpunit.de). 100% code coverage is not required, but all significant parts should have unit tests.
171
-
172
-
Make sure that the full suite of tests passes before committing changes by running:
173
-
174
-
phing test
175
-
176
-
### Building
177
-
178
-
You can build the magedbm2 phar archive with [Box](https://github.com/box-project/box2) by running:
179
-
180
-
phing build
181
-
182
-
### Releasing
183
-
184
-
To release a new version of magedbm2:
116
+
magedbm2 view-config
117
+
118
+
The `view-config` command will show you the result of the merged configuration files for debugging.
185
119
186
-
1. Build the phar archive
187
-
2. Update the `manifest.json` file with the new version information
120
+
# Contributing
188
121
189
-
- Use `https://github.com/space48/magedbm2/releases/download/<version>/magedbm2.phar` as the URL
190
-
191
-
- To calculate the sha1 checksum of the phar archive, run: `sha1sum magedbm2.phar`
122
+
For information on how to contribute to the project, see:
192
123
193
-
3. Update the Installation instructions above with the new version
194
-
4. Create a release on [Github](https://github.com/space48/magedbm2/releases)
As well as stripping certain tables in your projects, you might also have project-specific columns containing sensitive data that you do not want exported. To do this you need to define which formatter to apply to the column.
4
+
5
+
Depending on the type of entity that you want to anonymise you'll either need to use the `tables` or the `eav` key in the configuration. Assuming that we've added a new table called `super_pay` with two columns that need anonymisation and we've added a new attribute to customers.
You have access to the [Faker](https://github.com/fzaninotto/Faker) library.
19
+
20
+
The format of the formatter string is either `${className}` if the class implements `Meanbee\Magedbm2\Anonymizer\FormatterInterface` or `${className}::${method}` if you want to call a method directly.
Magedbm2 looks for configuration files in a number of places before merging them all together to build a final configuration structure. This allows developers to store various pieces of information in appropriate places. The credentials for AWS, for example, should be unique to each developer and so exist as a file on each developer's machine. The bucket and region to store the database backups in though should travel around with the code and be committed to the repository.
4
+
5
+
The load order of the configuration files are as follows:
6
+
7
+
***Dist**: Configuration distributed inside of the phar
8
+
* This is the base configuration that contains the majority of the default configuration such as the standard table groups. It can be found at `etc/config.yml`, in the root of the repository.
9
+
***User-specific**: User's configuration file
10
+
* This is a configuration file that could contain things like a user's AWS credentials and will be looked for in the user's home directory at `~/.magedbm2/config.yml`.
11
+
* The location of this file can be overwritten by passing a file name with the `--config` option.
12
+
***Project-specific**: Project configuration file
13
+
* This is the configuration file that could contain the project specific configuration that will be shipped around to each developer and environment by virtue of it living in the project repository. This will be loaded from `.magedbm2.yml` in the current working directory.
14
+
* The location of this file can be overwritten by passing a file name with the `--project-config` option.
Often in your projects you'll have a list of tables that you want to strip as a group. You may be familiar with this concept in [magerun](https://github.com/netz98/n98-magerun) when passing `--strip=@development` to the `db:dump` command. Magedbm2 ships with a default set of strip table groups but you can add your by adding a new key under `table-groups` in your configuration, e.g.
4
+
5
+
table-groups:
6
+
- id: superpay
7
+
description: Sensitive Super Pay tables
8
+
tables: superpay_* super_logs superpay
9
+
10
+
You can then use that table group in your `put` command, `magedbm2 put --strip=@superpay myproject`.
0 commit comments