diff --git a/.gitignore b/.gitignore index b08068b..1046f4d 100644 --- a/.gitignore +++ b/.gitignore @@ -2,5 +2,8 @@ configs test/fixtures/test*.sql pkg +.redcar .rvmrc .bundle +*.out +rdoc diff --git a/CHANGELOG b/CHANGELOG new file mode 100644 index 0000000..f45f18d --- /dev/null +++ b/CHANGELOG @@ -0,0 +1,19 @@ +0.2.2 Feb 13, 2012 +=== + +* merge contributions from: James Coleman, Aaron Peckham, James Tippett +* breaking change: note that "supress*" configuration parameters now use correct spelling. Existing config files you may have will need housekeeping. +* new configuration options: suppress_sequence_update; suppress_indexes; use_timezones +* adds a Gemfile to help setup development environment + +0.1.1 May 3, 2011 +=== + +* updated for pg driver ~> 0.11.0 +* add a check for mysql cmd, e.g. mysql or mysql5 +* remove deprecated feature per gem -v 1.7.2 + +0.1.0 September 18, 2010 +=== + +* Initial packaging as a gem \ No newline at end of file diff --git a/Gemfile b/Gemfile index c4cf727..cbc1bb7 100644 --- a/Gemfile +++ b/Gemfile @@ -1,6 +1,8 @@ source :rubygems -gem 'mysql' -gem 'pg' -gem 'test-unit' -gem 'jeweler' +group :development do + gem "bundler", "~> 1.0.21" + gem "jeweler", "~> 1.5.2" +end + +gemspec diff --git a/Gemfile.lock b/Gemfile.lock index 5a0a12f..da2950f 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,21 +1,34 @@ +PATH + remote: . + specs: + mysql2psql (0.2.0) + mysql (= 2.8.1) + mysql2psql + pg (~> 0.11.0) + GEM remote: http://rubygems.org/ specs: git (1.2.5) - jeweler (1.6.0) + jeweler (1.5.2) bundler (~> 1.0.0) git (>= 1.2.5) rake + json (1.6.5) mysql (2.8.1) pg (0.11.0) - rake (0.8.7) - test-unit (2.3.0) + rake (0.9.2.2) + rdoc (3.12) + json (~> 1.4) + test-unit (2.4.7) PLATFORMS ruby DEPENDENCIES - jeweler - mysql - pg - test-unit + bundler (~> 1.0.21) + jeweler (~> 1.5.2) + mysql2psql! + rake (~> 0.9.2.2) + rdoc (~> 3.12) + test-unit (>= 2.1.1) diff --git a/README.rdoc b/README.rdoc index 2d61d4b..97ba3df 100644 --- a/README.rdoc +++ b/README.rdoc @@ -11,9 +11,8 @@ to contact me, I'll help you. Mysql2psql is packaged as a gem. Install as usual (use sudo if required). $ gem install mysql2psql - NB: the gem hasn't been formally released on http://rubygems.org/ yet. For now you need to clone the git repo at http://github.com/tardate/mysql2postgres and 'rake install' -After installation, the "mysql2psql" command will be available. +After installation, the "mysql2psql" command will be available. When run, it will generate a default configuration file in the current directory if a config file is not already found. The default configuration filename is mysql2psql.yml. @@ -42,83 +41,93 @@ After editing the configuration file and launching tool, you will see something == Database driver dependencies -Mysql2psql uses the 'mysql' and 'pg' gems for database connectivity. +Mysql2psql uses the 'mysql' and 'pg' gems for database connectivity. Mysql2psql gem installation should automatically install these too. If you encounter any issues with db connectivity, verify that the 'mysql' and 'pg' gems are installed and working correctly first. The 'mysql' gem in particular may need a hint on where to find the mysql headers and libraries: $ [sudo] gem install mysql -- --with-mysql-dir=/usr/local/mysql - -NB: With Ruby 1.8.x, the gem install will usually complain about "No definition for ..." errors. These are non-fatal and just affect the documentation install. This doesn't happen with Ruby 1.9.2. - -With bundler you can do a simple: bundle install +NB: With Ruby 1.8.x, the gem install will usually complain about "No definition for ..." errors. These are non-fatal and just affect the documentation install. This doesn't happen with Ruby 1.9.2. + +== Contributing to Mysql2psql + +* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet +* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it +* Fork the project +* Start a feature/bugfix branch +* Commit and push until you are happy with your contribution +* Make sure to add tests for it. This is important so no-one unintentionally breaks it in a future version. +* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so the gem maintainers can cherry-pick around it. +* Send a pull request + == Getting the source -Mysql2psql was first produced by Max Lapshin who maintains the master -repository at http://github.com/maxlapshin/mysql2postgres +Mysql2psql was first produced by Max Lapshin who maintains the master +repository at https://github.com/maxlapshin/mysql2postgres -The gem packaged version you are currently looking at has yet to be merged into the master repo. The gem bundling and refactoring for testing was done by Paul Gallagher and -the repository is at http://github.com/tardate/mysql2postgres +the repository is at https://github.com/tardate/mysql2postgres +Gem packaging is done with Jeweler. Note that on windows this means you must run under mingw or other shell that supports git to do the gem bundling (but you can gem install the built gem in the normal Windows shell). +== Setting up for development + +The project includes a Gemfile so with bundler you can install gem dependencies with: + + gem install bundler # if not already installed + bundle install + +Note comments 'Database driver dependencies' regarding issues that you may encounter with the mysql gem. + == Running tests If you fork/clone the project, you will want to run the test suite (and add to it if you are developing new features or fixing bugs). - + A suite of tests are provided and are run using rake (rake -T for more info) rake test:units rake test:integration - rake test + rake test Rake without parameters (or "rake test") will run both the unit and integration tests. Unit tests are small standalone tests of the mysql2psql codebase that have no external dependencies (like a database) -Integration tests require suitable mysql and postgres databases to be setup in advance. +Integration tests require suitable mysql and postgres databases to be setup in advance. Running the integration tests *will* rewrite data and schema for the "mysql2psql_test" databases in mysql and postgres. Don't use this database for anything else! -mysql on localhost:3306 -- database created called "mysql2psql_test" -- user setup for "mysql2psql" with no password -- e.g. +Setting up mysql on localhost:3306 +- create a database called "mysql2psql_test" +- setup a user "mysql2psql" with no password + +e.g. + mysqladmin -uroot -p create mysql2psql_test mysql -uroot -p -e "grant all on mysql2psql_test.* to 'mysql2psql'@'localhost';" - # verify connecction: + # verify connection: mysql -umysql2psql -e "select database(),'yes' as connected" mysql2psql_test - -postgres on localhost:5432 -- database created called "mysql2psql_test" -- role (user) access setup for "mysql2psql" with no password -- e.g. + + +Setting up PostgreSQL on localhost:5432 +- create a database called "mysql2psql_test" +- setup a role (user) "mysql2psql" with no password + +e.g. + psql postgres -c "create role mysql2psql with login" psql postgres -c "create database mysql2psql_test with owner mysql2psql encoding='UTF8'" # verify connection: psql mysql2psql_test -U mysql2psql -c "\c" -== Notes, Limitations, Outstanding Issues.. -Todo: -- more tests -- release gem -- a windows cmd shim +== Notes, Limitations, Outstanding Issues.. -=== note from mgkimsal -I'm still having trouble with bit(1)/boolean fields -workaround I've found is to put output in file -then in VIM on file, search/replace the true/false binary fields with t/f -specifically +* more test coverage, as always... -(reversed on 3/23 - should be ^A gets f) -:%s/^@/t/g -:%s/^A/f/g -keystrokes are ctrl-v ctrl-shift-@ to get the 'true' binary field -keystrokes are ctrl-v ctrl-shift-A to get the 'false' binary field == Contributors Project founded by Max Lapshin -Other contributors (in git log order): +Contributors (roughly git log order): - Anton Ageev - Samuel Tribehou - Marco Nenciarini @@ -130,8 +139,10 @@ Other contributors (in git log order): - Jacob Coby - Neszt Tibor - Miroslav Kratochvil -- Paul Gallagher - - - - +- {Paul Gallagher}[https://github.com/tardate] +- {James Coleman}[https://github.com/jcoleman] +- {Aaron Peckham}[https://github.com/apeckham] +- {James Tippett}[https://github.com/jtippett] +- {Tim Morgan}[https://github.com/seven1m] +- {dakhota}[https://github.com/dakhota] +- {Matthew Soldo}[https://github.com/mattsoldo] diff --git a/Rakefile b/Rakefile index c7193f3..bae4717 100644 --- a/Rakefile +++ b/Rakefile @@ -13,7 +13,7 @@ begin gem.description = %Q{It can create postgresql dump from mysql database or directly load data from mysql to postgresql (at about 100 000 records per minute). Translates most data types and indexes.} gem.email = "gallagher.paul@gmail.com" - gem.homepage = "http://github.com/tardate/mysql2postgresql" + gem.homepage = "https://github.com/tardate/mysql2postgres" gem.authors = [ "Max Lapshin ", "Anton Ageev ", @@ -27,11 +27,19 @@ begin "Jacob Coby ", "Neszt Tibor ", "Miroslav Kratochvil ", - "Paul Gallagher " - ] + "Paul Gallagher ", + "James Coleman ", + "Aaron Peckham", + "James Tippett", + "Tim Morgan", + "dakhota", + "Matthew Soldo" + ] gem.add_dependency "mysql", "= 2.8.1" - gem.add_dependency "pg", "= 0.9.0" + gem.add_dependency "pg", "~> 0.11.0" gem.add_development_dependency "test-unit", ">= 2.1.1" + gem.add_development_dependency "rake", "~> 0.9.2.2" + gem.add_development_dependency "rdoc", "~> 3.12" # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings end Jeweler::GemcutterTasks.new @@ -63,9 +71,11 @@ end begin require 'rcov/rcovtask' Rcov::RcovTask.new do |test| - test.libs << 'test' + test.libs << 'lib' + test.libs << 'test/lib' test.pattern = 'test/**/*test.rb' test.verbose = true + test.rcov_opts << "--exclude gems/*" end rescue LoadError task :rcov do @@ -73,15 +83,13 @@ rescue LoadError end end - task :default => :test -require 'rake/rdoctask' -Rake::RDocTask.new do |rdoc| +require 'rdoc/task' +RDoc::Task.new do |rdoc| version = Mysql2psql::Version::STRING rdoc.rdoc_dir = 'rdoc' rdoc.title = "mysql2psql #{version}" - rdoc.rdoc_files.include('README*') - rdoc.rdoc_files.include('lib/**/*.rb') + rdoc.rdoc_files.include("README*", "lib/**/*.rb") end diff --git a/lib/mysql2psql/config.rb b/lib/mysql2psql/config.rb index 07f6c69..124aaa3 100644 --- a/lib/mysql2psql/config.rb +++ b/lib/mysql2psql/config.rb @@ -1,13 +1,13 @@ require 'mysql2psql/config_base' class Mysql2psql - + class Config < ConfigBase - + def initialize(configfilepath, generate_default_if_not_found = true) unless File.exists?(configfilepath) reset_configfile(configfilepath) if generate_default_if_not_found - if File.exists?(configfilepath) + if File.exists?(configfilepath) raise Mysql2psql::ConfigurationFileInitialized.new("\n No configuration file found. A new file has been initialized at: #{configfilepath} @@ -27,24 +27,34 @@ def reset_configfile(filepath) file.close end - def self.template(to_filename = nil, include_tables = [], exclude_tables = [], supress_data = false, supress_ddl = false, force_truncate = false) + # Returns template file text given +options+ hash which may include: + # :to_filename - default: nil + # :include_tables - default: [] + # :exclude_tables - default: [] + # :suppress_data - default: false + # :suppress_ddl - default: false + # :suppress_sequence_update - default: false + # :suppress_indexes - default: false + # :force_truncate - default: false + # :use_timezones - default: false + def self.template(options={}) configtext = <0 configtext += "\ntables:\n" include_tables.each do |t| @@ -65,36 +76,67 @@ def self.template(to_filename = nil, include_tables = [], exclude_tables = [], s #- table4 EOS + exclude_tables = options[:exclude_tables] || [] if exclude_tables.length>0 configtext += "\nexclude_tables:\n" exclude_tables.each do |t| configtext += "- #{t}\n" end end - if !supress_data.nil? + + if !options[:suppress_data].nil? + configtext += < @use_timezones}) + end unless @suppress_ddl _time2 = Time.now tables.each do |table| - writer.truncate(table) if force_truncate && supress_ddl + writer.truncate(table) if force_truncate && !suppress_ddl writer.write_contents(table, reader) - end unless @supress_data - + end unless @suppress_data + _time3 = Time.now tables.each do |table| - writer.write_indexes(table) - end unless @supress_ddl + writer.write_indexes(table) unless @suppress_indexes + end unless @suppress_ddl tables.each do |table| writer.write_constraints(table) - end unless @supress_ddl - - + end unless @suppress_ddl + + writer.close _time4 = Time.now puts "Table creation #{((_time2 - _time1) / 60).round} min, loading #{((_time3 - _time2) / 60).round} min, indexing #{((_time4 - _time3) / 60).round} min, total #{((_time4 - _time1) / 60).round} min" return 0 rescue => e $stderr.puts "Mysql2psql: conversion failed: #{e.to_s}" + $stderr.puts e.backtrace return -1 end end diff --git a/lib/mysql2psql/mysql_reader.rb b/lib/mysql2psql/mysql_reader.rb index 089ec4f..364e250 100644 --- a/lib/mysql2psql/mysql_reader.rb +++ b/lib/mysql2psql/mysql_reader.rb @@ -28,12 +28,16 @@ def columns def convert_type(type) case type - when /int.* unsigned/ + when /^int.* unsigned/ "bigint" when /bigint/ "bigint" when "bit(1)" "boolean" + when /smallint.* unsigned/ + "integer" + when /smallint/ + "smallint" when "tinyint(1)" "boolean" when /tinyint/ @@ -44,13 +48,15 @@ def convert_type(type) "varchar" when /char/ "char" - when /(float|decimal)/ + when /decimal/ "decimal" - when /double/ - "double precision" + when /float/ + "float" + when /real|double/ + "double precision" else type - end + end end def load_columns @@ -60,6 +66,7 @@ def load_columns fields = [] @reader.mysql.query("EXPLAIN `#{name}`") do |res| while field = res.fetch_row do + length = -1 length = field[1][/\((\d+)\)/, 1] if field[1] =~ /\((\d+)\)/ length = field[1][/\((\d+),(\d+)\)/, 1] if field[1] =~ /\((\d+),(\d+)\)/ desc = { @@ -88,7 +95,7 @@ def load_columns def indexes load_indexes unless @indexes - @indexes + @indexes end def foreign_keys @@ -147,7 +154,7 @@ def count_rows end def has_id? - !!columns.find {|col| col[:name] == "id"} + !!columns.find {|col| col[:name] == "id"} end def count_for_pager @@ -175,9 +182,9 @@ def reconnect end def initialize(options) - @host, @user, @passwd, @db, @port, @sock, @flag = - options.mysqlhostname('localhost'), options.mysqlusername, - options.mysqlpassword, options.mysqldatabase, + @host, @user, @passwd, @db, @port, @sock, @flag = + options.mysqlhostname('localhost'), options.mysqlusername, + options.mysqlpassword, options.mysqldatabase, options.mysqlport, options.mysqlsocket @port = nil if @port == "" # for things like Amazon's RDS you don't have a port and connect fails with "" for a value @sock = nil if @sock == "" @@ -186,9 +193,22 @@ def initialize(options) end attr_reader :mysql - + + def views + unless defined? @views + @mysql.query("SELECT t.TABLE_NAME FROM INFORMATION_SCHEMA.TABLES t WHERE t.TABLE_SCHEMA = '#{@db}' AND t.TABLE_TYPE = 'VIEW';") do |res| + @views = [] + res.each { |row| @views << row[0] } + end + end + + @views + end + def tables - @tables ||= @mysql.list_tables.map {|table| Table.new(self, table)} + @tables ||= (@mysql.list_tables - views).map do |table| + Table.new(self, table) + end end def paginated_read(table, page_size) @@ -207,4 +227,4 @@ def paginated_read(table, page_size) end end -end \ No newline at end of file +end diff --git a/lib/mysql2psql/postgres_db_writer.rb b/lib/mysql2psql/postgres_db_writer.rb index c64732f..72bb15d 100644 --- a/lib/mysql2psql/postgres_db_writer.rb +++ b/lib/mysql2psql/postgres_db_writer.rb @@ -8,8 +8,8 @@ class PostgresDbWriter < PostgresWriter attr_reader :conn, :hostname, :login, :password, :database, :schema, :port def initialize(options) - @hostname, @login, @password, @database, @port = - options.pghostname('localhost'), options.pgusername, + @hostname, @login, @password, @database, @port = + options.pghostname('localhost'), options.pgusername, options.pgpassword, options.pgdatabase, options.pgport(5432).to_s @database, @schema = database.split(":") open @@ -21,53 +21,61 @@ def open @conn.exec("SET client_encoding = 'UTF8'") @conn.exec("SET standard_conforming_strings = off") if @conn.server_version >= 80200 @conn.exec("SET check_function_bodies = false") - @conn.exec("SET client_min_messages = warning") + @conn.exec("SET client_min_messages = warning") end def close @conn.close end - + def exists?(relname) rc = @conn.exec("SELECT COUNT(*) FROM pg_class WHERE relname = '#{relname}'") (!rc.nil?) && (rc.to_a.length==1) && (rc.first.count.to_i==1) end - def write_table(table) + def write_sequence_update(table, options) + serial_key_column = table.columns.detect do |column| + column[:auto_increment] + end + + if serial_key_column + serial_key = serial_key_column[:name] + max_value = serial_key_column[:maxval].to_i < 1 ? 1 : serial_key_column[:maxval] + 1 + serial_key_seq = "#{table.name}_#{serial_key}_seq" + + if !options.supress_ddl + if @conn.server_version < 80200 + @conn.exec("DROP SEQUENCE #{serial_key_seq} CASCADE") if exists?(serial_key_seq) + else + @conn.exec("DROP SEQUENCE IF EXISTS #{serial_key_seq} CASCADE") + end + @conn.exec <<-EOF + CREATE SEQUENCE #{serial_key_seq} + INCREMENT BY 1 + NO MAXVALUE + NO MINVALUE + CACHE 1 + EOF + end + + if !options.supress_sequence_update + puts "Updated sequence #{serial_key_seq} to current value of #{max_value}" + @conn.exec sqlfor_set_serial_sequence(table, serial_key_seq, max_value) + end + end + end + + def write_table(table, options) puts "Creating table #{table.name}..." primary_keys = [] - serial_key = nil - maxval = nil columns = table.columns.map do |column| - if column[:auto_increment] - serial_key = column[:name] - maxval = column[:maxval].to_i < 1 ? 1 : column[:maxval] + 1 - end if column[:primary_key] primary_keys << column[:name] end - " " + column_description(column) + " " + column_description(column, options) end.join(",\n") - if serial_key - if @conn.server_version < 80200 - serial_key_seq = "#{table.name}_#{serial_key}_seq" - @conn.exec("DROP SEQUENCE #{serial_key_seq} CASCADE") if exists?(serial_key_seq) - else - @conn.exec("DROP SEQUENCE IF EXISTS #{table.name}_#{serial_key}_seq CASCADE") - end - @conn.exec <<-EOF - CREATE SEQUENCE #{table.name}_#{serial_key}_seq - INCREMENT BY 1 - NO MAXVALUE - NO MINVALUE - CACHE 1 - EOF - - @conn.exec "SELECT pg_catalog.setval('#{table.name}_#{serial_key}_seq', #{maxval}, true)" - end - if @conn.server_version < 80200 @conn.exec "DROP TABLE #{PGconn.quote_ident(table.name)} CASCADE;" if exists?(table.name) else @@ -83,34 +91,40 @@ def write_table(table) puts "Created table #{table.name}" end - + def write_indexes(table) puts "Indexing table #{table.name}..." if primary_index = table.indexes.find {|index| index[:primary]} - @conn.exec("ALTER TABLE #{PGconn.quote_ident(table.name)} ADD CONSTRAINT \"#{table.name}_pkey\" PRIMARY KEY(#{primary_index[:columns].map {|col| PGconn.quote_ident(col)}.join(", ")})") + index_sql = "ALTER TABLE #{PGconn.quote_ident(table.name)} ADD CONSTRAINT \"#{table.name}_pkey\" PRIMARY KEY(#{primary_index[:columns].map {|col| PGconn.quote_ident(col)}.join(", ")})" + @conn.exec(index_sql) end - + table.indexes.each do |index| next if index[:primary] unique = index[:unique] ? "UNIQUE " : nil - #MySQL allows an index name which could be equal to a table name, Postgres doesn't + # MySQL allows an index name which could be equal to a table name, Postgres doesn't indexname = index[:name] + indexname_quoted = '' + if indexname.eql?(table.name) - indexnamenew = "#{indexname}_index" - puts "WARNING: index \"#{indexname}\" equals table name. This is not allowed by postgres and will be renamed to \"#{indexnamenew}\"" - indexname = indexnamenew + indexname = (@conn.server_version < 90000) ? "#{indexname}_index" : nil + puts "WARNING: index \"#{index[:name]}\" equals table name. This is not allowed in PostgreSQL and will be renamed." end - - if @conn.server_version < 80200 - @conn.exec("DROP INDEX #{PGconn.quote_ident(indexname)} CASCADE;") if exists?(indexname) - else - @conn.exec("DROP INDEX IF EXISTS #{PGconn.quote_ident(indexname)} CASCADE;") + + if indexname + indexname_quoted = PGconn.quote_ident(indexname) + if @conn.server_version < 80200 + @conn.exec("DROP INDEX #{PGconn.quote_ident(indexname)} CASCADE;") if exists?(indexname) + else + @conn.exec("DROP INDEX IF EXISTS #{PGconn.quote_ident(indexname)} CASCADE;") + end end - @conn.exec("CREATE #{unique}INDEX #{PGconn.quote_ident(indexname)} ON #{PGconn.quote_ident(table.name)} (#{index[:columns].map {|col| PGconn.quote_ident(col)}.join(", ")});") + + index_sql = "CREATE #{unique}INDEX #{indexname_quoted} ON #{PGconn.quote_ident(table.name)} (#{index[:columns].map {|col| PGconn.quote_ident(col)}.join(", ")});" + @conn.exec(index_sql) end - - + #@conn.exec("VACUUM FULL ANALYZE #{PGconn.quote_ident(table.name)}") puts "Indexed table #{table.name}" rescue Exception => e @@ -118,7 +132,7 @@ def write_indexes(table) puts e puts e.backtrace[0,3].join("\n") end - + def write_constraints(table) table.foreign_keys.each do |key| key_sql = "ALTER TABLE #{PGconn.quote_ident(table.name)} ADD FOREIGN KEY (#{key[:column].map{|c|PGconn.quote_ident(c)}.join(', ')}) REFERENCES #{PGconn.quote_ident(key[:ref_table])}(#{key[:ref_column].map{|c|PGconn.quote_ident(c)}.join(', ')}) ON UPDATE #{key[:on_update]} ON DELETE #{key[:on_delete]}" @@ -149,7 +163,6 @@ def write_contents(table, reader) puts "Loading #{table.name}..." STDOUT.flush _counter = reader.paginated_read(table, 1000) do |row, counter| - line = [] process_row(table, row) @conn.put_copy_data(row.join("\t") + "\n") @@ -164,16 +177,25 @@ def write_contents(table, reader) if counter % 5000 == 0 @conn.put_copy_end + res = @conn.get_result + if res.cmdtuples != 5000 + puts "\nWARNING: #{table.name} expected 5000 tuple inserts got #{res.cmdtuples} at row #{counter}\n" + end @conn.exec(copy_line) end end - _time2 = Time.now - puts "\n#{_counter} rows loaded in #{((_time2 - _time1) / 60).round}min #{((_time2 - _time1) % 60).round}s" -# @conn.putline(".\n") @conn.put_copy_end + if _counter && (_counter % 5000) > 0 + res = @conn.get_result + if res.cmdtuples != (_counter % 5000) + puts "\nWARNING: table #{table.name} expected #{_counter % 5000} tuple inserts got #{res.cmdtuples}\n" + end + end + _time2 = Time.now + puts "\n#{table.name} #{_counter} rows loaded in #{((_time2 - _time1) / 60).round}min #{((_time2 - _time1) % 60).round}s" end - + end -end \ No newline at end of file +end diff --git a/lib/mysql2psql/postgres_file_writer.rb b/lib/mysql2psql/postgres_file_writer.rb index 4362e5c..3713d91 100644 --- a/lib/mysql2psql/postgres_file_writer.rb +++ b/lib/mysql2psql/postgres_file_writer.rb @@ -33,47 +33,58 @@ def truncate(table) EOF if serial_key @f << <<-EOF -SELECT pg_catalog.setval(pg_get_serial_sequence('#{table.name}', '#{serial_key}'), #{maxval}, true); +#{sqlfor_reset_serial_sequence(table,serial_key,maxval)} EOF end end - def write_table(table) - primary_keys = [] - serial_key = nil - maxval = nil - - columns = table.columns.map do |column| - if column[:auto_increment] - serial_key = column[:name] - maxval = column[:maxval].to_i < 1 ? 1 : column[:maxval] + 1 - end - if column[:primary_key] - primary_keys << column[:name] - end - " " + column_description(column) - end.join(",\n") + def write_sequence_update(table, options) + serial_key_column = table.columns.detect do |column| + column[:auto_increment] + end - if serial_key + if serial_key_column + serial_key = serial_key_column[:name] + serial_key_seq = "#{table.name}_#{serial_key}_seq" + max_value = serial_key_column[:maxval].to_i < 1 ? 1 : serial_key_column[:maxval] + 1 @f << <<-EOF -- --- Name: #{table.name}_#{serial_key}_seq; Type: SEQUENCE; Schema: public +-- Name: #{serial_key_seq}; Type: SEQUENCE; Schema: public -- +EOF + + if !options.supress_ddl + @f << <<-EOF +DROP SEQUENCE IF EXISTS #{serial_key_seq} CASCADE; -DROP SEQUENCE IF EXISTS #{table.name}_#{serial_key}_seq CASCADE; - -CREATE SEQUENCE #{table.name}_#{serial_key}_seq +CREATE SEQUENCE #{serial_key_seq} INCREMENT BY 1 NO MAXVALUE NO MINVALUE CACHE 1; - - -SELECT pg_catalog.setval('#{table.name}_#{serial_key}_seq', #{maxval}, true); - - EOF +EOF + end + + if !options.supress_sequence_update + @f << <<-EOF +#{sqlfor_set_serial_sequence(table, serial_key_seq, max_value)} +EOF + end end + end + + def write_table(table, options) + primary_keys = [] + serial_key = nil + maxval = nil + + columns = table.columns.map do |column| + if column[:primary_key] + primary_keys << column[:name] + end + " " + column_description(column, options) + end.join(",\n") @f << <<-EOF -- Table: #{table.name} @@ -126,7 +137,6 @@ def write_contents(table, reader) EOF reader.paginated_read(table, 1000) do |row, counter| - line = [] process_row(table, row) @f << row.join("\t") + "\n" end diff --git a/lib/mysql2psql/postgres_writer.rb b/lib/mysql2psql/postgres_writer.rb index 9609ed4..51df2ed 100644 --- a/lib/mysql2psql/postgres_writer.rb +++ b/lib/mysql2psql/postgres_writer.rb @@ -5,139 +5,164 @@ class Mysql2psql class PostgresWriter < Writer - def column_description(column) - "#{PGconn.quote_ident(column[:name])} #{column_type_info(column)}" + def column_description(column, options) + "#{PGconn.quote_ident(column[:name])} #{column_type_info(column, options)}" end - - def column_type(column) - column_type_info(column).split(" ").first - end - - def column_type_info(column) + + def column_type(column, options={}) if column[:auto_increment] - return "integer DEFAULT nextval('#{column[:table_name]}_#{column[:name]}_seq'::regclass) NOT NULL" + 'integer' + else + case column[:type] + when 'char' + "character(#{column[:length]})" + when 'varchar' + "character varying(#{column[:length]})" + when /tinyint|smallint/ + 'smallint' + when 'real', /float/, 'double precision' + 'double precision' + when 'decimal' + # TODO: seven1m thinks "real" instead? + "numeric(#{column[:length] || 10}, #{column[:decimals] || 5})" + when 'datetime', 'timestamp' + "timestamp with#{options[:use_timezones] ? '' : 'out'} time zone" + when 'time' + "time with#{options[:use_timezones] ? '' : 'out'} time zone" + when 'tinyblob', 'mediumblob', 'longblob', 'blob', 'varbinary' + 'bytea' + when 'tinytext', 'mediumtext', 'longtext', 'text' + 'text' + when /^enum/ + enum = column[:type].gsub(/enum|\(|\)/, '') + max_enum_size = enum.split(',').map{ |check| check.size() -2}.sort[-1] + "character varying(#{max_enum_size}) check( #{column[:name]} in (#{enum}))" + when 'integer', 'bigint', 'boolean', 'date' + column[:type] + else + puts "Unknown #{column.inspect}" + '' + end end + end - default = column[:default] ? " DEFAULT #{column[:default] == nil ? 'NULL' : "'"+PGconn.escape(column[:default])+"'"}" : nil - null = column[:null] ? "" : " NOT NULL" - type = - case column[:type] - - # String types - when "char" - default = default + "::char" if default - "character(#{column[:length]})" - when "varchar" - default = default + "::character varying" if default - # puts "VARCHAR: #{column.inspect}" - "character varying(#{column[:length]})" - - # Integer and numeric types - when "integer" - default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default].to_i}" if default - "integer" - when "bigint" - default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default].to_i}" if default - "bigint" - when "tinyint" - default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default].to_i}" if default - "smallint" - - when "boolean" - default = " DEFAULT #{column[:default].to_i == 1 ? 'true' : 'false'}" if default - "boolean" - when "float" - default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default].to_f}" if default - "real" - when "float unsigned" - default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default].to_f}" if default - "real" - when "decimal" - default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default]}" if default - "numeric(#{column[:length] || 10}, #{column[:decimals] || 0})" - - when "double precision" - default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default]}" if default - "double precision" - - # Mysql datetime fields - when "datetime" - default = nil - "timestamp without time zone" - when "date" - default = nil - "date" - when "timestamp" - default = " DEFAULT CURRENT_TIMESTAMP" if column[:default] == "CURRENT_TIMESTAMP" - default = " DEFAULT '1970-01-01 00:00'" if column[:default] == "0000-00-00 00:00" - default = " DEFAULT '1970-01-01 00:00:00'" if column[:default] == "0000-00-00 00:00:00" - "timestamp without time zone" - when "time" - default = " DEFAULT NOW()" if default - "time without time zone" + def column_default(column) + if column[:auto_increment] + "nextval('#{column[:table_name]}_#{column[:name]}_seq'::regclass)" + elsif column[:default] + case column[:type] + when 'char' + "'#{PGconn.escape(column[:default])}'::char" + when 'varchar', /^enum/ + "'#{PGconn.escape(column[:default])}'::character varying" + when 'integer', 'bigint', /tinyint|smallint/ + column[:default].to_i + when 'real', /float/ + column[:default].to_f + when 'decimal', 'double precision' + column[:default] + when 'boolean' + case column[:default] + when nil + 'NULL' + when 0, '0', "b'0'" + 'false' + else + # Case for 1, '1', "b'1'" (for BIT(1) the data type), or anything non-nil and non-zero (for the TINYINT(1) type) + 'true' + end + when 'timestamp', 'datetime', 'date' + case column[:default] + when 'CURRENT_TIMESTAMP' + 'CURRENT_TIMESTAMP' + when '0000-00-00' + "'1970-01-01'" + when '0000-00-00 00:00' + "'1970-01-01 00:00'" + when '0000-00-00 00:00:00' + "'1970-01-01 00:00:00'" + else + "'#{PGconn.escape(column[:default])}'" + end + when 'time' + "'#{PGconn.escape(column[:default])}'" + else + # TODO: column[:default] will never be nil here. + # Perhaps we should also issue a warning if this case is encountered. + "#{column[:default] == nil ? 'NULL' : "'"+PGconn.escape(column[:default])+"'"}" + end + end + end - when "tinyblob" - "bytea" - when "mediumblob" - "bytea" - when "longblob" - "bytea" - when "blob" - "bytea" - when "varbinary" - "bytea" - when "tinytext" - "text" - when "mediumtext" - "text" - when "longtext" - "text" - when "text" - "text" - when /^enum/ - default = default + "::character varying" if default - enum = column[:type].gsub(/enum|\(|\)/, '') - max_enum_size = enum.split(',').map{ |check| check.size() -2}.sort[-1] - "character varying(#{max_enum_size}) check( #{column[:name]} in (#{enum}))" + def column_type_info(column, options) + type = column_type(column, options) + if type + not_null = !column[:null] || column[:auto_increment] ? ' NOT NULL' : '' + default = column[:default] || column[:auto_increment] ? " DEFAULT #{column_default(column)}" : '' + "#{type}#{default}#{not_null}" else - puts "Unknown #{column.inspect}" - column[:type].inspect - return "" + '' end - "#{type}#{default}#{null}" end - - def process_row(table, row) - table.columns.each_with_index do |column, index| - if column[:type] == "time" + def process_row(table, row) + table.columns.each_with_index do |column, index| + if column[:type] == 'time' + begin row[index] = "%02d:%02d:%02d" % [row[index].hour, row[index].minute, row[index].second] + rescue + # Don't fail on nil date/time. end - - if row[index].is_a?(Mysql::Time) - row[index] = row[index].to_s.gsub('0000-00-00 00:00', '1970-01-01 00:00') - row[index] = row[index].to_s.gsub('0000-00-00 00:00:00', '1970-01-01 00:00:00') - end - - if column_type(column) == "boolean" - row[index] = row[index] == 1 ? 't' : row[index] == 0 ? 'f' : row[index] - end - - if row[index].is_a?(String) - if column_type(column) == "bytea" - row[index] = PGconn.escape_bytea(row[index]) + elsif row[index].is_a?(Mysql::Time) + row[index] = row[index].to_s.gsub('0000-00-00 00:00', '1970-01-01 00:00') + row[index] = row[index].to_s.gsub('0000-00-00 00:00:00', '1970-01-01 00:00:00') + elsif column[:type] == 'boolean' + row[index] = ( + case row[index] + when nil + '\N' # See note below about null values. + when 0, "\0" + 'f' + else + # Case for 1, "\1" (for the BIT(1) data type), or anything non-nil and non-zero (to handle the TINYINT(1) type) + 't' + end + ) + elsif row[index].is_a?(String) + if column_type(column) == "bytea" + row[index] = PGconn.escape_bytea(row[index]) + else + if row[index] == '\N' || row[index] == '\.' + row[index] = '\\' + row[index] # Escape our two PostgreSQL-text-mode-special strings. else - row[index] = row[index].gsub(/\\/, '\\\\\\').gsub(/\n/,'\n').gsub(/\t/,'\t').gsub(/\r/,'\r').gsub(/\0/, '') + # Awesome side-effect producing conditional. Don't do this at home. + unless row[index].gsub!(/\0/, '').nil? + puts "Removed null bytes from string since PostgreSQL TEXT types don't allow the storage of null bytes." + end + + row[index] = row[index].dump + row[index] = row[index].slice(1, row[index].size-2) end end - - row[index] = '\N' if !row[index] + elsif row[index].nil? + # Note: '\N' not "\N" is correct here: + # The string containing the literal backslash followed by 'N' + # represents database NULL value in PostgreSQL's text mode. + row[index] = '\N' end + end end - + def truncate(table) end - + + def sqlfor_set_serial_sequence(table, serial_key_seq, max_value) + "SELECT pg_catalog.setval('#{serial_key_seq}', #{max_value}, true);" + end + def sqlfor_reset_serial_sequence(table, serial_key, max_value) + "SELECT pg_catalog.setval(pg_get_serial_sequence('#{table.name}', '#{serial_key}'), #{max_value}, true);" + end + end -end \ No newline at end of file +end diff --git a/lib/mysql2psql/version.rb b/lib/mysql2psql/version.rb index c61d5c1..efb916e 100644 --- a/lib/mysql2psql/version.rb +++ b/lib/mysql2psql/version.rb @@ -1,7 +1,7 @@ class Mysql2psql module Version MAJOR = 0 - MINOR = 1 + MINOR = 2 PATCH = 0 STRING = [MAJOR, MINOR, PATCH].compact.join('.') diff --git a/mysql2psql.gemspec b/mysql2psql.gemspec index fb52551..f70fb13 100644 --- a/mysql2psql.gemspec +++ b/mysql2psql.gemspec @@ -1,15 +1,15 @@ # Generated by jeweler # DO NOT EDIT THIS FILE DIRECTLY -# Instead, edit Jeweler::Tasks in Rakefile, and run the gemspec command +# Instead, edit Jeweler::Tasks in Rakefile, and run 'rake gemspec' # -*- encoding: utf-8 -*- Gem::Specification.new do |s| s.name = %q{mysql2psql} - s.version = "0.1.0" + s.version = "0.2.0" s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version= - s.authors = ["Max Lapshin ", "Anton Ageev ", "Samuel Tribehou ", "Marco Nenciarini ", "James Nobis ", "quel ", "Holger Amann ", "Maxim Dobriakov ", "Michael Kimsal ", "Jacob Coby ", "Neszt Tibor ", "Miroslav Kratochvil ", "Paul Gallagher "] - s.date = %q{2010-09-19} + s.authors = ["Max Lapshin ", "Anton Ageev ", "Samuel Tribehou ", "Marco Nenciarini ", "James Nobis ", "quel ", "Holger Amann ", "Maxim Dobriakov ", "Michael Kimsal ", "Jacob Coby ", "Neszt Tibor ", "Miroslav Kratochvil ", "Paul Gallagher ", "James Coleman ", "Aaron Peckham", "James Tippett", "Tim Morgan", "dakhota", "Matthew Soldo"] + s.date = %q{2012-02-13} s.default_executable = %q{mysql2psql} s.description = %q{It can create postgresql dump from mysql database or directly load data from mysql to postgresql (at about 100 000 records per minute). Translates most data types and indexes.} @@ -19,73 +19,145 @@ Gem::Specification.new do |s| "README.rdoc" ] s.files = [ - ".gitignore", - "MIT-LICENSE", - "README.rdoc", - "Rakefile", - "bin/mysql2psql", - "lib/mysql2psql.rb", - "lib/mysql2psql/config.rb", - "lib/mysql2psql/config_base.rb", - "lib/mysql2psql/converter.rb", - "lib/mysql2psql/errors.rb", - "lib/mysql2psql/mysql_reader.rb", - "lib/mysql2psql/postgres_db_writer.rb", - "lib/mysql2psql/postgres_file_writer.rb", - "lib/mysql2psql/postgres_writer.rb", - "lib/mysql2psql/version.rb", - "lib/mysql2psql/writer.rb", - "mysql2psql.gemspec", - "test/fixtures/config_all_options.yml", - "test/fixtures/seed_integration_tests.sql", - "test/integration/convert_to_db_test.rb", - "test/integration/convert_to_file_test.rb", - "test/integration/converter_test.rb", - "test/integration/mysql_reader_base_test.rb", - "test/integration/mysql_reader_test.rb", - "test/integration/postgres_db_writer_base_test.rb", - "test/lib/ext_test_unit.rb", - "test/lib/test_helper.rb", - "test/units/config_base_test.rb", - "test/units/config_test.rb", - "test/units/postgres_file_writer_test.rb" + "CHANGELOG", + "Gemfile", + "Gemfile.lock", + "MIT-LICENSE", + "README.rdoc", + "Rakefile", + "bin/mysql2psql", + "lib/mysql2psql.rb", + "lib/mysql2psql/config.rb", + "lib/mysql2psql/config_base.rb", + "lib/mysql2psql/converter.rb", + "lib/mysql2psql/errors.rb", + "lib/mysql2psql/mysql_reader.rb", + "lib/mysql2psql/postgres_db_writer.rb", + "lib/mysql2psql/postgres_file_writer.rb", + "lib/mysql2psql/postgres_writer.rb", + "lib/mysql2psql/version.rb", + "lib/mysql2psql/writer.rb", + "mysql2psql.gemspec", + "test/fixtures/config_all_options.yml", + "test/fixtures/seed_integration_tests.sql", + "test/integration/convert_to_db_test.rb", + "test/integration/convert_to_file_test.rb", + "test/integration/converter_test.rb", + "test/integration/mysql_reader_base_test.rb", + "test/integration/mysql_reader_test.rb", + "test/integration/postgres_db_writer_base_test.rb", + "test/lib/ext_test_unit.rb", + "test/lib/test_helper.rb", + "test/units/config_base_test.rb", + "test/units/config_test.rb", + "test/units/postgres_file_writer_test.rb" ] - s.homepage = %q{http://github.com/tardate/mysql2postgresql} - s.rdoc_options = ["--charset=UTF-8"] + s.homepage = %q{https://github.com/tardate/mysql2postgres} s.require_paths = ["lib"] - s.rubygems_version = %q{1.3.7} + s.rubygems_version = %q{1.6.2} s.summary = %q{Tool for converting mysql database to postgresql} s.test_files = [ "test/integration/convert_to_db_test.rb", - "test/integration/convert_to_file_test.rb", - "test/integration/converter_test.rb", - "test/integration/mysql_reader_base_test.rb", - "test/integration/mysql_reader_test.rb", - "test/integration/postgres_db_writer_base_test.rb", - "test/lib/ext_test_unit.rb", - "test/lib/test_helper.rb", - "test/units/config_base_test.rb", - "test/units/config_test.rb", - "test/units/postgres_file_writer_test.rb" + "test/integration/convert_to_file_test.rb", + "test/integration/converter_test.rb", + "test/integration/mysql_reader_base_test.rb", + "test/integration/mysql_reader_test.rb", + "test/integration/postgres_db_writer_base_test.rb", + "test/lib/ext_test_unit.rb", + "test/lib/test_helper.rb", + "test/units/config_base_test.rb", + "test/units/config_test.rb", + "test/units/postgres_file_writer_test.rb" ] if s.respond_to? :specification_version then - current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION s.specification_version = 3 if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then + s.add_runtime_dependency(%q, [">= 0"]) + s.add_development_dependency(%q, ["~> 1.0.21"]) + s.add_development_dependency(%q, ["~> 1.5.2"]) + s.add_development_dependency(%q, ["~> 1.0.21"]) + s.add_development_dependency(%q, ["~> 1.5.2"]) + s.add_development_dependency(%q, ["~> 1.0.21"]) + s.add_development_dependency(%q, ["~> 1.5.2"]) + s.add_development_dependency(%q, [">= 2.1.1"]) + s.add_development_dependency(%q, ["~> 0.9.2.2"]) + s.add_development_dependency(%q, ["~> 3.12"]) + s.add_development_dependency(%q, [">= 2.1.1"]) + s.add_development_dependency(%q, ["~> 0.9.2.2"]) + s.add_development_dependency(%q, ["~> 3.12"]) + s.add_development_dependency(%q, [">= 2.1.1"]) + s.add_development_dependency(%q, ["~> 0.9.2.2"]) + s.add_development_dependency(%q, ["~> 3.12"]) + s.add_development_dependency(%q, [">= 2.1.1"]) + s.add_development_dependency(%q, ["~> 0.9.2.2"]) + s.add_development_dependency(%q, ["~> 3.12"]) + s.add_development_dependency(%q, [">= 2.1.1"]) + s.add_development_dependency(%q, ["~> 0.9.2.2"]) + s.add_development_dependency(%q, ["~> 3.12"]) s.add_runtime_dependency(%q, ["= 2.8.1"]) - s.add_runtime_dependency(%q, ["= 0.9.0"]) + s.add_runtime_dependency(%q, ["~> 0.11.0"]) s.add_development_dependency(%q, [">= 2.1.1"]) + s.add_development_dependency(%q, ["~> 0.9.2.2"]) + s.add_development_dependency(%q, ["~> 3.12"]) else + s.add_dependency(%q, [">= 0"]) + s.add_dependency(%q, ["~> 1.0.21"]) + s.add_dependency(%q, ["~> 1.5.2"]) + s.add_dependency(%q, ["~> 1.0.21"]) + s.add_dependency(%q, ["~> 1.5.2"]) + s.add_dependency(%q, ["~> 1.0.21"]) + s.add_dependency(%q, ["~> 1.5.2"]) + s.add_dependency(%q, [">= 2.1.1"]) + s.add_dependency(%q, ["~> 0.9.2.2"]) + s.add_dependency(%q, ["~> 3.12"]) + s.add_dependency(%q, [">= 2.1.1"]) + s.add_dependency(%q, ["~> 0.9.2.2"]) + s.add_dependency(%q, ["~> 3.12"]) + s.add_dependency(%q, [">= 2.1.1"]) + s.add_dependency(%q, ["~> 0.9.2.2"]) + s.add_dependency(%q, ["~> 3.12"]) + s.add_dependency(%q, [">= 2.1.1"]) + s.add_dependency(%q, ["~> 0.9.2.2"]) + s.add_dependency(%q, ["~> 3.12"]) + s.add_dependency(%q, [">= 2.1.1"]) + s.add_dependency(%q, ["~> 0.9.2.2"]) + s.add_dependency(%q, ["~> 3.12"]) s.add_dependency(%q, ["= 2.8.1"]) - s.add_dependency(%q, ["= 0.9.0"]) + s.add_dependency(%q, ["~> 0.11.0"]) s.add_dependency(%q, [">= 2.1.1"]) + s.add_dependency(%q, ["~> 0.9.2.2"]) + s.add_dependency(%q, ["~> 3.12"]) end else + s.add_dependency(%q, [">= 0"]) + s.add_dependency(%q, ["~> 1.0.21"]) + s.add_dependency(%q, ["~> 1.5.2"]) + s.add_dependency(%q, ["~> 1.0.21"]) + s.add_dependency(%q, ["~> 1.5.2"]) + s.add_dependency(%q, ["~> 1.0.21"]) + s.add_dependency(%q, ["~> 1.5.2"]) + s.add_dependency(%q, [">= 2.1.1"]) + s.add_dependency(%q, ["~> 0.9.2.2"]) + s.add_dependency(%q, ["~> 3.12"]) + s.add_dependency(%q, [">= 2.1.1"]) + s.add_dependency(%q, ["~> 0.9.2.2"]) + s.add_dependency(%q, ["~> 3.12"]) + s.add_dependency(%q, [">= 2.1.1"]) + s.add_dependency(%q, ["~> 0.9.2.2"]) + s.add_dependency(%q, ["~> 3.12"]) + s.add_dependency(%q, [">= 2.1.1"]) + s.add_dependency(%q, ["~> 0.9.2.2"]) + s.add_dependency(%q, ["~> 3.12"]) + s.add_dependency(%q, [">= 2.1.1"]) + s.add_dependency(%q, ["~> 0.9.2.2"]) + s.add_dependency(%q, ["~> 3.12"]) s.add_dependency(%q, ["= 2.8.1"]) - s.add_dependency(%q, ["= 0.9.0"]) + s.add_dependency(%q, ["~> 0.11.0"]) s.add_dependency(%q, [">= 2.1.1"]) + s.add_dependency(%q, ["~> 0.9.2.2"]) + s.add_dependency(%q, ["~> 3.12"]) end end diff --git a/test/fixtures/config_all_options.yml b/test/fixtures/config_all_options.yml index 06d0b83..b8b366a 100644 --- a/test/fixtures/config_all_options.yml +++ b/test/fixtures/config_all_options.yml @@ -1,10 +1,10 @@ mysql: hostname: localhost port: 3306 - socket: /tmp/mysql.sock + socket: username: somename password: secretpassword - database: somename + database: somename destination: # if file is given, output goes to file, else postgres @@ -28,11 +28,23 @@ exclude_tables: - table5 - table6 -# if supress_data is true, only the schema definition will be exported/migrated, and not the data -supress_data: true +# if suppress_data is true, only the schema definition will be exported/migrated, and not the data +suppress_data: true -# if supress_ddl is true, only the data will be exported/imported, and not the schema -supress_ddl: false +# if suppress_ddl is true, only the data will be exported/imported, and not the schema +suppress_ddl: false + +# if suppress_sequence_update is true, the sequences for serial (auto-incrementing) columns +# will not be update to the current maximum value of that column in the database +# if suppress_ddl is not set to true, then this option is implied to be false as well (unless overridden here) +suppress_sequence_update: false + +# if suppress_indexes is true, indexes will not be exported/migrated. +suppress_indexes: false # if force_truncate is true, forces a table truncate before table loading force_truncate: false + +# if use_timezones is true, timestamp/time columns will be created in postgres as "with time zone" +# rather than "without time zone" +use_timezones: false diff --git a/test/fixtures/seed_integration_tests.sql b/test/fixtures/seed_integration_tests.sql index 8163255..579e988 100644 --- a/test/fixtures/seed_integration_tests.sql +++ b/test/fixtures/seed_integration_tests.sql @@ -4,21 +4,116 @@ DROP TABLE IF EXISTS numeric_types_basics; CREATE TABLE numeric_types_basics ( id int, f_tinyint TINYINT, + f_tinyint_u TINYINT UNSIGNED, f_smallint SMALLINT, + f_smallint_u SMALLINT UNSIGNED, f_mediumint MEDIUMINT, + f_mediumint_u MEDIUMINT UNSIGNED, f_int INT, + f_int_u INT UNSIGNED, f_integer INTEGER, + f_integer_u INTEGER UNSIGNED, f_bigint BIGINT, + f_bigint_u BIGINT UNSIGNED, f_real REAL, f_double DOUBLE, f_float FLOAT, + f_float_u FLOAT UNSIGNED, f_decimal DECIMAL, f_numeric NUMERIC ); INSERT INTO numeric_types_basics VALUES -(1,1,1,1,1,1,1,1,1,1,1,1), -(2,2,2,2,2,2,2,2,2,2,2,2), -(23,23,23,23,23,23,23,23,23,23,23,23); +( 1, 2, 3, 4, 5, 6, 7, 8, 9,10,11,12,13,14,15,16,17,18,19), +( 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1), +( 3,23,23,23,23,23,23,23,23,23,23,23,23,23,23,23,23,23,23), +( 4, -128, 0,-32768, 0,-8388608, 0,-2147483648, 0,-2147483648, 0,-9223372036854775808, 0, 1, 1, 1, 1, 1, 1), +( 5, 127, 255, 32767, 65535, 8388607, 16777215, 2147483647, 4294967295, 2147483647, 4294967295, 9223372036854775807, 18446744073709551615, 1, 1, 1, 1, 1, 1); + +DROP TABLE IF EXISTS basic_autoincrement; +CREATE TABLE basic_autoincrement ( + auto_id INT(11) NOT NULL AUTO_INCREMENT, + auto_dummy INT, + PRIMARY KEY (auto_id) +); + +INSERT INTO basic_autoincrement(auto_dummy) VALUES +(1),(2),(23); + +-- see GH#22 float conversion error +DROP TABLE IF EXISTS numeric_type_floats; +CREATE TABLE numeric_type_floats ( + latitude FLOAT, + longitude FLOAT +); + +INSERT INTO numeric_type_floats(latitude,longitude) VALUES +(1.1,2.2); + +-- see GH#18 smallint error +DROP TABLE IF EXISTS gh18_smallint; +CREATE TABLE gh18_smallint ( + s_smallint SMALLINT, + u_smallint SMALLINT UNSIGNED +); + +INSERT INTO gh18_smallint(s_smallint,u_smallint) VALUES +(-32768,32767), +(-1,0), +(32767,65535); + +-- see https://github.com/maxlapshin/mysql2postgres/issues/27 +DROP TABLE IF EXISTS test_boolean_conversion; +CREATE TABLE test_boolean_conversion ( + test_name VARCHAR(25), + bit_1 BIT(1), + tinyint_1 TINYINT(1), + bit_1_default_0 BIT(1) DEFAULT 0, + bit_1_default_1 BIT(1) DEFAULT 1, + tinyint_1_default_0 TINYINT(1) DEFAULT 0, + tinyint_1_default_1 TINYINT(1) DEFAULT 1, + tinyint_1_default_2 TINYINT(1) DEFAULT 2 -- Test the fact that 1 byte isn't limited to [0,1] +); + +INSERT INTO test_boolean_conversion (test_name, bit_1, tinyint_1) +VALUES ('test-null', NULL, NULL), + ('test-false', 0, 0), + ('test-true', 1, 1); +INSERT INTO test_boolean_conversion (test_name, tinyint_1) VALUES ('test-true-nonzero', 2); + +CREATE OR REPLACE VIEW test_view AS +SELECT b.test_name +FROM test_boolean_conversion b; + +DROP TABLE IF EXISTS test_null_conversion; +CREATE TABLE test_null_conversion (column_a VARCHAR(10)); +INSERT INTO test_null_conversion (column_a) VALUES (NULL); + +DROP TABLE IF EXISTS test_datetime_conversion; +CREATE TABLE test_datetime_conversion ( + column_a DATETIME, + column_b TIMESTAMP, + column_c DATETIME DEFAULT '0000-00-00', + column_d DATETIME DEFAULT '0000-00-00 00:00', + column_e DATETIME DEFAULT '0000-00-00 00:00:00', + column_f TIME +); +INSERT INTO test_datetime_conversion (column_a, column_f) VALUES ('0000-00-00 00:00', '08:15:30'); + +DROP TABLE IF EXISTS test_index_conversion; +CREATE TABLE test_index_conversion (column_a VARCHAR(10)); +CREATE UNIQUE INDEX index_test_index_conversion_on_column_a ON test_index_conversion (column_a); + +DROP TABLE IF EXISTS test_foreign_keys_child; +DROP TABLE IF EXISTS test_foreign_keys_parent; +CREATE TABLE test_foreign_keys_parent (id INT NOT NULL, PRIMARY KEY (id)) ENGINE=INNODB; +CREATE TABLE test_foreign_keys_child (id INT, test_foreign_keys_parent_id INT, + INDEX par_ind (test_foreign_keys_parent_id), + FOREIGN KEY (test_foreign_keys_parent_id) REFERENCES test_foreign_keys_parent(id) ON DELETE CASCADE +) ENGINE=INNODB; + +DROP TABLE IF EXISTS test_enum; +CREATE TABLE test_enum (name ENUM('small', 'medium', 'large')); +INSERT INTO test_enum (name) VALUES ('medium'); \ No newline at end of file diff --git a/test/integration/convert_to_db_test.rb b/test/integration/convert_to_db_test.rb index 249c45e..f47abef 100644 --- a/test/integration/convert_to_db_test.rb +++ b/test/integration/convert_to_db_test.rb @@ -4,26 +4,134 @@ class ConvertToDbTest < Test::Unit::TestCase - class << self - def startup - seed_test_database - @@options=get_test_config_by_label(:localmysql_to_db_convert_all) - @@mysql2psql = Mysql2psql.new([@@options.filepath]) - @@mysql2psql.convert - @@mysql2psql.writer.open - end - def shutdown - @@mysql2psql.writer.close - delete_files_for_test_config(@@options) - end - end def setup + $stdout = StringIO.new + $stderr = StringIO.new + + seed_test_database + @options=get_test_config_by_label(:localmysql_to_db_convert_all) + @mysql2psql = Mysql2psql.new([@options.filepath]) + @mysql2psql.convert + @mysql2psql.writer.open end + def teardown + @mysql2psql.writer.close + delete_files_for_test_config(@options) + + $stdout = STDOUT + $stderr = STDERR + end + + def exec_sql_on_psql(sql, parameters=nil) + @mysql2psql.writer.conn.exec(sql, parameters) + end + + def get_boolean_test_record(name) + exec_sql_on_psql('SELECT * FROM test_boolean_conversion WHERE test_name = $1', [name]).first end def test_table_creation - assert_true @@mysql2psql.writer.exists?('numeric_types_basics') + assert_true @mysql2psql.writer.exists?('numeric_types_basics') + assert_true @mysql2psql.writer.exists?('basic_autoincrement') + assert_true @mysql2psql.writer.exists?('numeric_type_floats') + end + + def test_boolean_conversion_to_true + true_record = get_boolean_test_record('test-true') + assert_equal 't', true_record['bit_1'] + assert_equal 't', true_record['tinyint_1'] + + true_nonzero_record = get_boolean_test_record('test-true-nonzero') + assert_equal 't', true_nonzero_record['tinyint_1'] end + def test_boolean_conversion_to_false + false_record = get_boolean_test_record('test-false') + assert_equal 'f', false_record['bit_1'] + assert_equal 'f', false_record['tinyint_1'] + end + + def test_boolean_conversion_of_null + null_record = get_boolean_test_record('test-null') + assert_nil null_record['bit_1'] + assert_nil null_record['tinyint_1'] + end + + def test_null_conversion + result = exec_sql_on_psql('SELECT column_a FROM test_null_conversion').first + assert_nil result['column_a'] + end + + def test_datetime_conversion + result = exec_sql_on_psql('SELECT column_a, column_f FROM test_datetime_conversion').first + assert_equal '1970-01-01 00:00:00', result['column_a'] + assert_equal '08:15:30', result['column_f'] + end + + def test_datetime_defaults + result = exec_sql_on_psql(<<-SQL) + SELECT a.attname, + pg_catalog.format_type(a.atttypid, a.atttypmod), + (SELECT substring(pg_catalog.pg_get_expr(d.adbin, d.adrelid) for 128) + FROM pg_catalog.pg_attrdef d + WHERE d.adrelid = a.attrelid AND d.adnum = a.attnum AND a.atthasdef) AS default + FROM pg_catalog.pg_attribute a + WHERE a.attrelid = 'test_datetime_conversion'::regclass AND a.attnum > 0 + SQL + + assert_equal 6, result.count + + result.each do |row| + if row["attname"] == "column_f" + assert_equal "time without time zone", row["format_type"] + else + assert_equal "timestamp without time zone", row["format_type"] + end + + case row["attname"] + when "column_a" + assert_nil row["default"] + when "column_b" + assert_equal "now()", row["default"] + when "column_c", "column_d", "column_e" + assert_equal "'1970-01-01 00:00:00'::timestamp without time zone", row["default"] + end + end + end + + def test_index_conversion + result = exec_sql_on_psql(%( + SELECT pg_get_indexdef(indexrelid) + FROM pg_index + join pg_class on pg_class.oid=pg_index.indrelid + where pg_class.relname='test_index_conversion' + )).first + assert_equal "CREATE UNIQUE INDEX index_test_index_conversion_on_column_a ON test_index_conversion USING btree (column_a)", result["pg_get_indexdef"] + end + + def test_foreign_keys + result = exec_sql_on_psql("SELECT conname, pg_catalog.pg_get_constraintdef(r.oid, true) as condef FROM pg_catalog.pg_constraint r WHERE r.conrelid = 'test_foreign_keys_child'::regclass") + expected = {"condef" => "FOREIGN KEY (test_foreign_keys_parent_id) REFERENCES test_foreign_keys_parent(id) ON UPDATE RESTRICT ON DELETE CASCADE", "conname" => "test_foreign_keys_child_test_foreign_keys_parent_id_fkey"} + assert_equal expected, result.first + end + + def test_output + $stdout.rewind + actual = $stdout.read + + assert_match /Counting rows of test_foreign_keys_child/, actual + end + + def test_enum + result = exec_sql_on_psql(<<-SQL) + SELECT r.conname, pg_catalog.pg_get_constraintdef(r.oid, true) + FROM pg_catalog.pg_constraint r + WHERE r.conrelid = 'test_enum'::regclass AND r.contype = 'c' + ORDER BY 1 + SQL + + assert_equal 1, result.count + assert_equal "CHECK (name::text = ANY (ARRAY['small'::character varying, 'medium'::character varying, 'large'::character varying]::text[]))", result.first["pg_get_constraintdef"] + end end \ No newline at end of file diff --git a/test/integration/convert_to_file_test.rb b/test/integration/convert_to_file_test.rb index f7c440c..77516d7 100644 --- a/test/integration/convert_to_file_test.rb +++ b/test/integration/convert_to_file_test.rb @@ -4,63 +4,118 @@ class ConvertToFileTest < Test::Unit::TestCase - class << self - def startup - seed_test_database - @@options=get_test_config_by_label(:localmysql_to_file_convert_all) - @@mysql2psql = Mysql2psql.new([@@options.filepath]) - @@mysql2psql.convert - @@content = IO.read(@@mysql2psql.options.destfile) - end - def shutdown - delete_files_for_test_config(@@options) - end - end def setup + seed_test_database + @options=get_test_config_by_label(:localmysql_to_file_convert_all) + @mysql2psql = Mysql2psql.new([@options.filepath]) + @mysql2psql.convert + @content = IO.read(@mysql2psql.options.destfile) end def teardown + delete_files_for_test_config(@options) end def content - @@content + @content end - + + # verify table creation def test_table_creation assert_not_nil content.match('DROP TABLE IF EXISTS "numeric_types_basics" CASCADE') assert_not_nil content.match(/CREATE TABLE "numeric_types_basics"/) end + # tests for the conversion of numeric types + def get_basic_numerics_match(column) + Regexp.new('CREATE TABLE "numeric_types_basics".*"' + column + '" ([^\n]*).*\)', Regexp::MULTILINE).match( content ) + end def test_basic_numerics_tinyint - assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_tinyint" smallint,.*\)', Regexp::MULTILINE).match( content ) + match = get_basic_numerics_match( 'f_tinyint' ) + assert_match /smallint/,match[1] + end + def test_basic_numerics_tinyint_u + match = get_basic_numerics_match( 'f_tinyint_u' ) + assert_match /smallint/,match[1] end def test_basic_numerics_smallint - assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_smallint" integer,.*\)', Regexp::MULTILINE).match( content ) + match = get_basic_numerics_match( 'f_smallint' ) + assert_match /smallint/,match[1] + end + def test_basic_numerics_smallint_u + match = get_basic_numerics_match( 'f_smallint_u' ) + assert_match /integer/,match[1] end def test_basic_numerics_mediumint - assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_mediumint" integer,.*\)', Regexp::MULTILINE).match( content ) + match = get_basic_numerics_match( 'f_mediumint' ) + assert_match /integer/,match[1] end def test_basic_numerics_int - assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_int" integer,.*\)', Regexp::MULTILINE).match( content ) + match = get_basic_numerics_match( 'f_int' ) + assert_match /integer/,match[1] end def test_basic_numerics_integer - assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_integer" integer,.*\)', Regexp::MULTILINE).match( content ) + match = get_basic_numerics_match( 'f_integer' ) + assert_match /integer/,match[1] end def test_basic_numerics_bigint - assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_bigint" bigint,.*\)', Regexp::MULTILINE).match( content ) + match = get_basic_numerics_match( 'f_bigint' ) + assert_match /bigint/,match[1] + #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_bigint" bigint,.*\)', Regexp::MULTILINE).match( content ) end def test_basic_numerics_real - assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_real" double precision,.*\)', Regexp::MULTILINE).match( content ) + match = get_basic_numerics_match( 'f_real' ) + assert_match /double precision/,match[1] + #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_real" double precision,.*\)', Regexp::MULTILINE).match( content ) end def test_basic_numerics_double - assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_double" double precision,.*\)', Regexp::MULTILINE).match( content ) + match = get_basic_numerics_match( 'f_double' ) + assert_match /double precision/,match[1] + #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_double" double precision,.*\)', Regexp::MULTILINE).match( content ) end def test_basic_numerics_float - assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_float" numeric\(20, 0\),.*\)', Regexp::MULTILINE).match( content ) + match = get_basic_numerics_match( 'f_float' ) + assert_match /double precision/,match[1] + #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_float" double precision.*\)', Regexp::MULTILINE).match( content ) + end + def test_basic_numerics_float_u + match = get_basic_numerics_match( 'f_float_u' ) + assert_match /double precision/,match[1] + #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_float_u" double precision.*\)', Regexp::MULTILINE).match( content ) end def test_basic_numerics_decimal - assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_decimal" numeric\(10, 0\),.*\)', Regexp::MULTILINE).match( content ) + match = get_basic_numerics_match( 'f_decimal' ) + assert_match /numeric/,match[1] + #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_decimal" numeric\(10, 0\),.*\)', Regexp::MULTILINE).match( content ) end def test_basic_numerics_numeric - assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_numeric" numeric\(10, 0\)[\w\n]*\)', Regexp::MULTILINE).match( content ) + match = get_basic_numerics_match( 'f_numeric' ) + assert_match /numeric/,match[1] + #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_numeric" numeric\(10, 0\)[\w\n]*\)', Regexp::MULTILINE).match( content ) end + # test boolean conversion + def get_boolean_column_definition_match(column) + Regexp.new('CREATE TABLE "test_boolean_conversion".*"' + column + '" ([^\n]*)[^;]*', Regexp::MULTILINE).match(content)[1] + end + def test_boolean_default_values + assert_match /DEFAULT false/, get_boolean_column_definition_match('bit_1_default_0') + assert_match /DEFAULT false/, get_boolean_column_definition_match('tinyint_1_default_0') + + assert_match /DEFAULT true/, get_boolean_column_definition_match('bit_1_default_1') + assert_match /DEFAULT true/, get_boolean_column_definition_match('tinyint_1_default_1') + assert_match /DEFAULT true/, get_boolean_column_definition_match('tinyint_1_default_2') + end + + # test autoincrement handling + def test_autoincrement + assert_not_nil Regexp.new('CREATE TABLE "basic_autoincrement".*"auto_id" integer DEFAULT.*\)', Regexp::MULTILINE).match( content ) + end + + # test view creation (or lack thereof!) + def test_should_not_copy_views_as_tables + assert_no_match /CREATE TABLE "test_view"/, content + end + + def test_truncate + assert_match /TRUNCATE/, content + end end \ No newline at end of file diff --git a/test/integration/converter_test.rb b/test/integration/converter_test.rb index a362d9d..df3947a 100644 --- a/test/integration/converter_test.rb +++ b/test/integration/converter_test.rb @@ -4,18 +4,12 @@ class ConverterTest < Test::Unit::TestCase - class << self - def startup - seed_test_database - @@options=get_test_config_by_label(:localmysql_to_file_convert_nothing) - end - def shutdown - delete_files_for_test_config(@@options) - end - end def setup + seed_test_database + @@options=get_test_config_by_label(:localmysql_to_file_convert_nothing) end def teardown + delete_files_for_test_config(@@options) end def options @@options diff --git a/test/integration/mysql_reader_base_test.rb b/test/integration/mysql_reader_base_test.rb index 2616d88..0769ca0 100644 --- a/test/integration/mysql_reader_base_test.rb +++ b/test/integration/mysql_reader_base_test.rb @@ -4,39 +4,31 @@ class MysqlReaderBaseTest < Test::Unit::TestCase - class << self - def startup - seed_test_database - @@options = get_test_config_by_label(:localmysql_to_file_convert_nothing) - end - def shutdown - delete_files_for_test_config(@@options) - end - end def setup + seed_test_database + @options = get_test_config_by_label(:localmysql_to_file_convert_nothing) end + def teardown - end - def options - @@options + delete_files_for_test_config(@options) end def test_mysql_connection assert_nothing_raised do - reader = Mysql2psql::MysqlReader.new(options) + reader = Mysql2psql::MysqlReader.new(@options) end end def test_mysql_reconnect assert_nothing_raised do - reader = Mysql2psql::MysqlReader.new(options) + reader = Mysql2psql::MysqlReader.new(@options) reader.reconnect end end def test_mysql_connection_without_port assert_nothing_raised do - options.mysqlport = "" - options.mysqlsocket = "" - reader = Mysql2psql::MysqlReader.new(options) + @options.mysqlport = "" + @options.mysqlsocket = "" + reader = Mysql2psql::MysqlReader.new(@options) end end end \ No newline at end of file diff --git a/test/integration/mysql_reader_test.rb b/test/integration/mysql_reader_test.rb index 87c1be3..e904ed6 100644 --- a/test/integration/mysql_reader_test.rb +++ b/test/integration/mysql_reader_test.rb @@ -2,22 +2,16 @@ class MysqlReaderTest < Test::Unit::TestCase - class << self - def startup - seed_test_database - @@options = get_test_config_by_label(:localmysql_to_file_convert_nothing) - @@reader=get_test_reader(@@options) - end - def shutdown - delete_files_for_test_config(@@options) - end - end def setup + seed_test_database + @options = get_test_config_by_label(:localmysql_to_file_convert_nothing) + @reader=get_test_reader(@options) end def teardown + delete_files_for_test_config(@options) end def reader - @@reader + @reader end def test_db_connection @@ -31,7 +25,7 @@ def test_tables_collection assert_equal 'numeric_types_basics', values[0].name end def test_paginated_read - expected_rows=3 + expected_rows=5 page_size=2 expected_pages=(1.0 * expected_rows / page_size).ceil diff --git a/test/integration/postgres_db_writer_base_test.rb b/test/integration/postgres_db_writer_base_test.rb index 8e25129..56d0fa8 100644 --- a/test/integration/postgres_db_writer_base_test.rb +++ b/test/integration/postgres_db_writer_base_test.rb @@ -4,26 +4,18 @@ class PostgresDbWriterBaseTest < Test::Unit::TestCase - class << self - def startup - seed_test_database - @@options=get_test_config_by_label(:localmysql_to_db_convert_nothing) - end - def shutdown - delete_files_for_test_config(@@options) - end - end def setup + seed_test_database + @options = get_test_config_by_label(:localmysql_to_db_convert_nothing) end + def teardown - end - def options - @@options + delete_files_for_test_config(@options) end def test_pg_connection assert_nothing_raised do - reader = Mysql2psql::PostgresDbWriter.new(options) + reader = Mysql2psql::PostgresDbWriter.new(@options) end end diff --git a/test/lib/ext_test_unit.rb b/test/lib/ext_test_unit.rb index 025e35e..1e1eabb 100644 --- a/test/lib/ext_test_unit.rb +++ b/test/lib/ext_test_unit.rb @@ -26,5 +26,8 @@ def assert_false(object, message="") def assert_true(object, message="") assert_equal(true, object, message) end + def assert_nil(object, message="") + assert_equal(nil, object, message) + end end diff --git a/test/lib/test_helper.rb b/test/lib/test_helper.rb index ea49633..f8257fb 100644 --- a/test/lib/test_helper.rb +++ b/test/lib/test_helper.rb @@ -12,7 +12,8 @@ def seed_test_database options=get_test_config_by_label(:localmysql_to_file_convert_nothing) seedfilepath = "#{File.dirname(__FILE__)}/../fixtures/seed_integration_tests.sql" - rc=system("mysql -u#{options.mysqlusername} #{options.mysqldatabase} < #{seedfilepath}") + mysql_cmd = `which mysql`.empty? ? 'mysql5' : 'mysql' + rc=system("#{mysql_cmd} -u#{options.mysqlusername} #{options.mysqldatabase} < #{seedfilepath}") raise StandardError unless rc return true rescue @@ -25,7 +26,7 @@ def get_test_reader(options) require 'mysql2psql/mysql_reader' Mysql2psql::MysqlReader.new(options) rescue - raise StandardError.new("Failed to initialize integration test db. See README for setup requirements.") + raise StandardError.new("Failed to initialize integration test db. See README for setup requirements.") end def get_test_file_writer(options) @@ -53,12 +54,13 @@ def get_temp_file(basename) path end - -def get_new_test_config(to_file = true, include_tables = [], exclude_tables = [], supress_data = false, supress_ddl = false, force_truncate = false) +def get_new_test_config(options={}) require 'mysql2psql/config' require 'mysql2psql/config_base' - to_filename = to_file ? get_temp_file('mysql2psql_tmp_output') : nil - configtext = Mysql2psql::Config.template(to_filename, include_tables, exclude_tables, supress_data, supress_ddl, force_truncate) + if options.delete(:to_file) + options[:to_filename] = get_temp_file('mysql2psql_tmp_output') + end + configtext = Mysql2psql::Config.template(options) configfile=get_temp_file('mysql2psql_tmp_config') File.open(configfile, 'w') {|f| f.write(configtext) } Mysql2psql::ConfigBase.new( configfile ) @@ -67,18 +69,59 @@ def get_new_test_config(to_file = true, include_tables = [], exclude_tables = [] end def get_test_config_by_label(name) - case name + options = case name when :localmysql_to_file_convert_nothing - get_new_test_config(true, ['unobtainium'], ['kryptonite'], true, true, false) + { + :to_file => true, + :include_tables => ['unobtainium'], + :exclude_tables => ['kryptonite'], + :suppress_data => true, + :suppress_ddl => true, + :supress_sequence_update => false, + :suppress_indexes => false, + :force_truncate => false, + :use_timezones => false + } when :localmysql_to_file_convert_all - get_new_test_config(true, [], [], false, false, false) + { + :to_file => true, + :include_tables => [], + :exclude_tables => [], + :suppress_data => false, + :suppress_ddl => false, + :supress_sequence_update => false, + :suppress_indexes => false, + :force_truncate => true, + :use_timezones => false + } when :localmysql_to_db_convert_all - get_new_test_config(false, [], [], false, false, false) + { + :to_file => false, + :include_tables => [], + :exclude_tables => [], + :suppress_data => false, + :suppress_ddl => false, + :supress_sequence_update => false, + :suppress_indexes => false, + :force_truncate => false, + :use_timezones => false + } when :localmysql_to_db_convert_nothing - get_new_test_config(false, ['unobtainium'], ['kryptonite'], true, true, false) + { + :to_file => false, + :include_tables => ['unobtainium'], + :exclude_tables => ['kryptonite'], + :suppress_data => true, + :suppress_ddl => true, + :supress_sequence_update => false, + :suppress_indexes => false, + :force_truncate => false, + :use_timezones => false + } else raise StandardError.new("Invalid label: #{name}") end + get_new_test_config(options) end def delete_files_for_test_config(config) diff --git a/test/units/config_test.rb b/test/units/config_test.rb index 5d531f6..d0133ef 100644 --- a/test/units/config_test.rb +++ b/test/units/config_test.rb @@ -23,9 +23,51 @@ def test_config_file_not_found value = Mysql2psql::Config.new(configfile_not_found, false) end end + def test_initialize_new_config_file assert_raise(Mysql2psql::ConfigurationFileInitialized) do value = Mysql2psql::Config.new(configfile_new, true) end end + + def test_config_option_pgdatabase_as_array_index + expected = 'somename' + config = Mysql2psql::Config.new(configfile_all_opts, false) + assert_equal expected,config[:pgdatabase] + end + def test_template_option_to_filename + expected = 'test_filename' + value = Mysql2psql::Config.template({ :to_filename => expected }) + assert_match /file: #{expected}/,value + end + def test_template_option_suppress_data + expected = true + value = Mysql2psql::Config.template({ :suppress_data => expected }) + assert_match /suppress_data: #{expected}/,value #NB: option spelling needs fixing + end + def test_template_option_suppress_ddl + expected = true + value = Mysql2psql::Config.template({ :suppress_ddl => expected }) + assert_match /suppress_ddl: #{expected}/,value #NB: option spelling needs fixing + end + def test_template_option_suppress_sequence_update + expected = true + value = Mysql2psql::Config.template({ :suppress_sequence_update => expected }) + assert_match /suppress_sequence_update: #{expected}/,value #NB: option spelling needs fixing + end + def test_template_option_suppress_indexes + expected = true + value = Mysql2psql::Config.template({ :suppress_indexes => expected }) + assert_match /suppress_indexes: #{expected}/,value #NB: option spelling needs fixing + end + def test_template_option_force_truncate + expected = true + value = Mysql2psql::Config.template({ :force_truncate => expected }) + assert_match /force_truncate: #{expected}/,value + end + def test_template_option_use_timezones + expected = true + value = Mysql2psql::Config.template({ :use_timezones => expected }) + assert_match /use_timezones: #{expected}/,value + end end \ No newline at end of file