![Antonin Delpeuch](/assets/img/avatar_default.png)
* Take snapshot of docs for version 3.4 * Versioning for docs of the cross function, for #2504 * Document 'Store archive file' option (#1963) * Remove unsupported preference from 3.4 docs (#2624) * Mention that forEach works on JSON objects (#3149) * Remove wholeText from 3.4 docs (#3180) * Document -H, /H CLI options (#3288) * Migrate Wikibase documentation from Wikidata (#1640) * Miscellanous, copy-editing
12 KiB
id | title | sidebar_label |
---|---|---|
migrating-older-extensions | Migrating older Extensions | Migrating older Extensions |
Migrating from Ant to Maven
Why are we doing this change?
Ant is a fairly old (antique?) build system that does not incorporate any dependency management. By migrating to Maven we are making it easier for developers to extend OpenRefine with new libraries, and stop having to ship dozens of .jar files in the repository. Using the Maven repository also encourages developers to add dependencies to released versions of libraries instead of custom snapshots that are hard to update.
When was this change made?
The migration was done between 3.0 and 3.1-beta with this commit:
47323a9e75
How to migrate an extension
You will need to write a pom.xml
in the root folder of your extension to configure the compilation process with Maven. Sample pom.xml
files for extensions can be found in the extensions that are shipped with OpenRefine (gdata
, database
, jython
, pc-axis
and wikidata
). A sample extension (sample
) is also provided, with a minimal build file.
For any library that your extension depends on, you should try to find a matching artifact in the Maven Central repository. If you can find such an artifact, delete the .jar
file from your extension and add the dependency in your pom.xml
file. If you cannot find such an artifact, it is still possible to incorporate your own .jar
file using maven-install-plugin
that you can configure in your pom.xml
file as follows:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
<executions>
<execution>
<id>install-wdtk-datamodel</id>
<phase>process-resources</phase>
<configuration>
<file>${basedir}/lib/my-proprietary-library.jar</file>
<repositoryLayout>default</repositoryLayout>
<groupId>com.my.company</groupId>
<artifactId>my-library</artifactId>
<version>0.5.3-SNAPSHOT</version>
<packaging>jar</packaging>
<generatePom>true</generatePom>
</configuration>
<goals>
<goal>install-file</goal>
</goals>
</execution>
<!-- if you need to add more than one jar, add more execution blocks here -->
</executions>
</plugin>
And add the dependency to the <dependencies>
section as usual:
<dependency>
<groupId>com.my.company</groupId>
<artifactId>my-library</artifactId>
<version>0.5.3-SNAPSHOT</version>
</dependency>
Migrating to Wikimedia's i18n jQuery plugin
Why are we doing this change?
This adds various important localization features, such as the ability to handle plurals or interpolation. This also restores the language fallback (displaying strings in English if they are not available in the target language) which did not work with the previous set up.
When was the migration made?
The migration was made between 3.1-beta and 3.1, with this commit: 22322bd027
How to migrate an extension
You will need to update your translation files, merging nested objets in one global object, concatenating keys. You can do this by running the following Python script on all your JSON translation files:
import json
import sys
with open(sys.argv[1], 'r') as f:
j = json.loads(f.read())
result = {}
def translate(obj, path):
res = {}
if type(obj) == str:
result['/'.join(path)] = obj
else:
for k, v in obj.items():
new_path = path + [k]
translate(v, new_path)
translate(j, [])
with open(sys.argv[1], 'w') as f:
f.write(json.dumps(result, ensure_ascii=False, indent=4))
Then your javascript files which retrieve the translated strings should be updated: $.i18n._('core-dialogs')['cancel']
becomes $.i18n('core-dialogs/cancel')
. You can do this with the following sed
script:
sed -i "s/\$\.i18n._(['\"]\([A-Za-z0-9/_\\-]*\)['\"])\[['\"]\([A-Za-z0-9\-\_]*\)[\"']\]/$.i18n('\1\/\2')/g" my_javascript_file.js
You can then chase down the places where you are concatenating translated strings, and replace that with more flexible patterns using the plugin's features.
Migrating from org.json to Jackson
Why are we doing this change?
The org.json (or json-java) library has multiple drawbacks.
- First, it has limited functionality - all the serialization and deserialization has to be done explicitly - an important proportion of OpenRefine's code was dedicated to implementing these;
- Second, its implementation is not optimized for speed - multiple projects have reported speedups when migrating to more modern JSON libraries;
- Third, and this was the decisive factor to initiate the migration: its license is the MIT license with an additional condition which makes it non-free. Getting rid of this dependency was required by the Software Freedom Conservancy as a prerequisite to become a fiscal sponsor for the project.
When was the migration made?
This change was made between 3.1 and 3.2-beta, with this commit: 5639f1b2f1
How to migrate an extension or fork
You will need to use the Jackson library to serialize the classes that implement interfaces or extend classes exposed by OpenRefine.
The interface Jsonizable
was deleted. Any class that used to implement this now needs to be serializable by Jackson, producing the same format as the previous serialization code. This applies to any operation, facet, overlay model or GREL function. If you are new to Jackson, have a look at this tutorial to learn how to annotate your class for serialization. Once this is done, you can remove the void write(JSONWriter writer, Properties options)
method from your class. Note that it is important that you do this migration for all classes implementing the Jsonizable
interface that are exposed to OpenRefine's core.
We encourage you to migrate out of org.json completely, but this is only required for the classes that interact with OpenRefine's core.
General notes about migrating
OpenRefine's ObjectMapper is available at ParsingUtilities.mapper
. It is configured to only serialize the fields and getters that have been explicitly marked with @JsonProperty
(to avoid accidental JSON format changes due to refactoring). On deserialization it will ignore any field in the JSON payload that does not correspond to a field in the Java class. It has serializers and deserializers for OffsetDateTime
and LocalDateTime
.
Useful snippets to use in tests:
- deserialize an instance:
MyClass instance = ParsingUtilities.mapper.readValue(jsonString, MyClass.class);
(replaces calls toJsonizable.write
); - serialize an instance:
String json = ParsingUtilities.mapper.writeValueAsString(myInstance);
(replaces calls to static methods such asload
,loadStreaming
orreconstruct
); - the equivalent of
JSONObject
isObjectNode
, the equivalent ofJSONArray
isArrayNode
; - create an empty JSON object:
ParsingUtilities.mapper.createObjectNode()
(replacesnew JSONObject()
); - create an empty JSON array:
ParsingUtilities.mapper.createArrayNode()
(replacesnew JSONArray()
).
Before undertaking the migration, we recommend that you write some tests which serialize and deserialize your objects. This will help you make sure that the JSON format is preserved during the migration. One way to do this is to collect some sample JSON representations of your objects, and check in your tests that deserializing these JSON payloads and serializing them back to JSON preserves the JSON payload. Some utilities are available to help you with that in TestUtils
(we had some to test org.json serialization before we got rid of the dependency, feel free to copy them).
For functions
Before the migration, you had to explicitly define JSON serialization of functions with a write
method. You should now override the getters returning the various documentation fields.
Example: Cos
function before and after.
For operations
Before the JSON migration we refactored engine-dependent operations so that the engine configuration is represented by an EngineConfig
object instead of a JSONObject
. Therefore the constructor for your operation should be updated to use this new class. Your constructor should also be annotated to be used during deserialization.
Note that you do not need to explicitly serialize the operation type, this is already done for you by AbstractOperation
.
Example: ColumnRemovalOperation
before and after.
For changes
Changes are serialized in plain text but often relies on JSON serialization for parts of the data. Just use the methods above with ParsingUtilities.mapper
to maintain this behaviour.
Example: ReconChange
before and after.
For importers
The importing options have been migrated from JSONObject
to ObjectNode
. Your compiler should help you propagate this change. Utility functions in JSONUtilities
have been migrated to Jackson so you should have minimal changes if you used them.
Example: TabularImportingParserBase
before and after.
For overlay models
Migrate serialization and deserialization as for other objects.
Example: WikibaseSchema
before and after
For preference values
Any class that is stored in OpenRefine's preference now needs to implement the com.google.refine.preferences.PreferenceValue
interface. The static load
method and the write
method used previously for deserialization should be deleted and regular Jackson serialization and deserialization should be implemented instead. Note that you do not need to explicitly serialize the class name, this is already done for you by the interface.