It's working for me with FileMaker Pro Advanced 17 and Mathematica 12 (under macOS Mojave 10.14.4).
Did you install the jdbc driver that is included in the FileMaker Pro Advanced .dmg? It's in the Extras > xDBC > JDBC Client Driver folder.
Moreover, to connect from Mathematica, you already need to have FileMaker Pro Advanced running, of course, and to have the particular .fmp12 database open, too, so far as I can determine.
See https://fmhelp.filemaker.com/help/17/fmp/en/index.html#page/FMP_Help/odbc-jdbc.html