I dumped some data in mysql.sql format, the file size is more than 400MB. And change the file into Liquibase SQL changelog format
When using Liquibase maven plugin to load those data into database, get error below:
- Exception in thread "main" java.lang.reflect.InvocationTargetException
- at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
- at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
- at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
- at java.base/java.lang.reflect.Method.invoke(Method.java:566)
- at org.apache.maven.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:39)
- at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:122)
- at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:55)
- Caused by: liquibase.util.grammar.TokenMgrError: Lexical error at line 177, column 168779. Encountered: "\u0e34" (3636), after : ""
- at liquibase.util.grammar.SimpleSqlGrammarTokenManager.getNextToken(SimpleSqlGrammarTokenManager.java:1373)
- at liquibase.util.grammar.SimpleSqlGrammar.getNextToken(SimpleSqlGrammar.java:109)
- at liquibase.util.SqlParser.parse(SqlParser.java:40)
- at liquibase.util.StringUtils.processMutliLineSQL(StringUtils.java:55)
- at liquibase.change.AbstractSQLChange.generateStatements(AbstractSQLChange.java:220)
- at liquibase.database.AbstractJdbcDatabase.executeStatements(AbstractJdbcDatabase.java:1209)
- at liquibase.changelog.ChangeSet.execute(ChangeSet.java:600)
- at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:53)
- at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:83)
- at liquibase.Liquibase.update(Liquibase.java:202)
- at liquibase.Liquibase.update(Liquibase.java:179)
- at liquibase.Liquibase.update(Liquibase.java:334)
- at org.liquibase.maven.plugins.LiquibaseUpdate.doUpdate(LiquibaseUpdate.java:33)
- at org.liquibase.maven.plugins.AbstractLiquibaseUpdateMojo.performLiquibaseTask(AbstractLiquibaseUpdateMojo.java:30)
- at org.liquibase.maven.plugins.AbstractLiquibaseMojo.execute(AbstractLiquibaseMojo.java:366)
- at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
- at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
- at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
- at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
- at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
- at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
- at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
- at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
- at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
- at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
- at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
- at org.apache.maven.cli.MavenCli.execute(MavenCli.java:954)
- at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
- at org.apache.maven.cli.MavenCli.main(MavenCli.java:192)
- at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
- at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
- at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
- at java.base/java.lang.reflect.Method.invoke(Method.java:566)
- at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
- at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
- at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
- at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
- ... 7 more
Here is more information:
1. The file (*.mysql.sql) size is more than 400 MB, close to 500MB, with more than 2 million records
2. One INSERT sql statement might be very long in one line in the file, with about 1 million records. The example as below:
Not sure how many characters Liquibase/Java support to read in one line. Shall I separate those records into one record one INSERT statement?
- INSERT INTO `city` VALUES (241042,2090,'Xihe Nanzhai',40.150730,118.351120),(241043,2090,'Michengzhuang',40.064701,118.331512),(241044,2090,'Huangya',40.015282,118.354172),(241045,2090,'Nanguan',40.085850,118.278847),(241046,2090,'Jiahe',40.107140,118.343277),(241047,2090,'Xigang',40.097672,118.451134),(241048,2090,'Xinzhuang',40.038719,118.547623),(241049,2090,'Zhangguanying',40.018551,118.620064),(241050,2090,'Xinzhai',40.031429,118.698792),(241051,2090,'Panying',40.076981,118.661186),(241052,2090,'Yanjiadian',40.111111,118.682503),(241053,2090,'Tengzhuang',40.103649,118.698357),(241054,2090,'Lanruoyuan',40.064590,118.717552),(241055,2090,'Leigutai',40.161209,118.611450),(241056,2090,'Wuchong’an',40.178612,118.657784),(241057,2090,'Ximiming',40.149441,118.716942),(241058,2090,'Shangzhuangzi',40.166561,118.687866),(241059,2090,'Geluanhe',40.135349,118.651367),(241060,2090,'Shangjinshanyuan',40.143082,118.610268),(241061,2090,'Changlingfeng',40.172218,118.543243),(241062,2090,'Dongzhai',40.161201,118.490013),(241063,2090,'Xizhai',40.164280,118.458603),(241064,2090,'Erbozi',40.183620,118.448540),(241065,2090,'Gaoguzhuang',40.212601,118.506287),(241066,2090,'Miaolingtou',40.235458,118.426392),(241067,2090,'Qingshankou',40.293320,118.401451),(241068,2090,'Xiaying',40.317940,118.459030),(241069,2090,'Cuijia Buzi',40.344440,118.496391),(241070,2090,'Wafangzhuang',40.365292,118.425293),(241071,2090,'Hexi',40.510311,118.532150),(241072,2090,'Cuizhangzi',40.376942,118.662842),(241073,2090,'Qianzhangzi',40.379978,118.685822),(241074,2090,'Xiaomaping',40.279770,118.626373),(241075,2090,'Liubozi',40.274441,118.726669),(241076,2090,'Qingheyan',40.271832,118.586090),(241077,2090,'Dazhuang',40.182190,118.724091),(241078,2090,'Pinglinzhencun',40.120480,118.747093),(241079,2090,'Banbishan',40.525902,118.704880),(241080,2090,'Yangshugou',40.541920,118.701492),(241081,2090,'Shangbancheng',40.615131,118.672913),(241082,2090,'Fanzhangzi',40.629219,118.727440),(241083,2090,'Jianshanzi',40.622299,118.566422),(241084,2090,'Gangyaogou',40.634590,118.480820),(241085,2090,'Laoliangzi',40.665691,118.509567),(241086,2090,'Shangdian',40.674831,118.564568),(241087,2090,'Liushudixia',40.668652,118.604088),(241088,2090,'Yong’an',40.730419,118.583321),(241089,2090,'Liubadian',40.731628,118.603218),(241090,2090,'Dajikou',40.809200,118.577599),(241091,2090,'Shanzihou',40.812710,118.646812),(241092,2090,'Heishankou',40.887890,118.663528),(241093,2090,'Nanling',40.971531,118.664391),(241094,2090,'Shuiquan',40.882011,118.556961),(241095,2090,'Yanzhangzi',40.823780,118.522507),(241096,2090,'Xiaofanzhangzi',40.909611,118.316833),(241097,2090,'Xiyamenzi',40.918331,118.388611),(241098,2090,'Yanjiagou',40.955441,118.301842),(241099,2090,'Yezhuhe',40.892601,118.252991),(241100,2090,'Yujiayuan',40.781921,118.331886),(241101,2090,'Bajia',40.661942,118.262222),(241102,2090,'Sunjiazhuang',40.577839,118.348763),(241103,2090,'Dasangyuan',40.557350,118.371750),(241104,2090,'Goumen',40.608608,118.447220),(241105,2090,'Dongbingjiao',40.587318,118.443680),(241106,2090,'Songling',40.293030,118.269081),(241107,2090,'Daheshan',40.278542,118.268883),(241108,2090,'Pengzhuang',40.243610,118.258713),(241109,2090,'Shalingzi',40.155579,118.293472),(241110,2090,'Dazhai',40.171398,118.393761),(241111,2090,'Xiacaonian',40.315620,118.659187),(241112,2090,'Shangcaonian',40.326679,118.649658),(241113,2096,'Oukenhe',49.790661,124.801376),(241114,2096,'Bayan Ewenke Minzu',49.462292,124.766060),(241115,2090,'Magezhuang',40.075771,117.772148),(241116,2090,'Housuotun',40.067478,117.813309),(241117,2090,'Guogezhuang',40.090481,117.814888),(241118,2090,'Beiying',40.101292,117.843452),(241119,2090,'Nangang',40.092312,117.869034),(241120,2090,'Wanggezhuang',40.023689,117.850403),(241121,2090,'Dongliangzihe',40.031879,117.878708),(241122,2090,'Qianmaozhuang',40.066109,117.923889),(241123,2090,'Shangzhuangtun',40.033699,117.940514),(241124,2090,'Xiaofengtai',40.063190,117.834877),(241125,2090,'Zhongtan',40.049419,117.808632),(241126,2090,'Dongmenzhuang',40.033379,117.799263),(241127,2090,'Dama Gezhuang',40.014568,117.803383),(241128,2090,'Hanhezhuang',40.106941,117.778061),(241129,2090,'Bahuzhuang',40.145390,117.764259),(241130,2090,'Mengjiapu',40.168098,117.797310), ...... ......