* Add utility functions to check/convert dates
* Add date tests and refactor to DRY up
* Fix date import - fixes#1908
Change from java.util.Date to OpenRefine 3.0+'s OffsetDateTime
Fixes#1908
* Centralize date conversion
* Moving utility methods to ParsingUtilities
* Fix tests
* Use standard text normalization - fixes#2898Fixes#2898. Fixes#409. Refs #650
Replaces homegrown ISO Latin-1 only character subsitition
with standard Java Normalize to NFD, followed by diacritic
removal and a few custom character expansions/replacements.
* Fix Mac build
* Improve compatibility with previous code
One intentional change is folding O with stroke to
oe instead of o.
- Use more powerful NFKD instead of NFD
- strip punctuation after decomposition since it can generate
new punctuation
- Add compatibility test for old asciify() method
- Add some graphically similar characters to substitution table
* Add oe character/ligature & more long S forms
* More tests for ligatures and Latin Extended
* Add Latin-1 Supplement tests
Fixes#1161
This change parallels what was done in #12571da3c00 to fix
the FingerprintKeyer and moves the diacritic removal before
the deduping. Includes a test.
* Truncate any completely empty columns on the right
Fixes#565
The current versions of Open Office create default spreadsheets
with over 1000 empty columns. Keep track of the rightmost
non-empty column when importing and truncate everything else.
Also adds a basic ODS import test.
* Fix dates in ODS spreadsheets
Fixes#2224
* Performance optimized version of ToNumber
Approximately 5x faster for floats (data dependent)
and about the same speed for integers.
- Instead of blindly trying to parse as Long, do a quick check
for obvious problems (e.g. decimal point).
- Don't trim. It's already done by called methods.
- Use valueOf() instead of parse() to avoid object creation
* Add Java Microbenchmark Harness
The shaded JAR is missing the OpenRefine classes, for a reason
that I haven't figured out, so requires openrefine-main.jar at runtime.
* Remove old implementations of ToNumber
* Remove unneeded dependencies from main project
* Clean up and reformat
Refs #2863
The tree importer sorts columns/column groups by how populated
they are, which is of arguable utility, but the tie-breaker
of ordering by shortest column name is completely silly.
This change removes that and, in conjunction with a stable sort
algorithm, will preserve the original order of the columns.
* Fix two deprecated methods usages
* Test ToNumber conversions
* Test behavior of all functions when passed 0 or 8 arguments
There are 16 which fail currently on 0 args (return null or
False instead of EvalError), but have been whitelisted until
we can verify whether it's safe to change them without introducing
compatibility issues.
There are 19 which fail to return an error on too many (ie 8) args.
No issue.
- we don't support Excel95, but make sure that it generates an exception
- move the test data file into the appropriate directory
- for any normal test, consider exceptions a failure
Fixes#565
The current versions of Open Office create default spreadsheets
with over 1000 empty columns. Keep track of the rightmost
non-empty column when importing and truncate everything else.
Also adds a basic ODS import test.
Fixes#2824
Versions up through 3.14.0 appear to work, but since odfdom bundles
Jena 3.9.0, we're going to be conservative and match that.
As an added bonus, includes a blank node test which will trigger
the failure.
* Fix charset encoding & MIME type handling
Character set (ie what we call "encoding") is part of the Content-Type,
*not* the Content-Encoding, which specifies compression (e.g. gzip).
This correctly sets the character set encoding as well as cleaning
the MIME type so that additional parsing doesn't need to be done
downstream (and removes that code).
* Use "text" instead of "text/line-based" as default fallback format
The TextLineBasedGuesser only tries a limited number of
formats (CSV, TSV, fixed), so we can't get out of that hole to
find JSON, XML, etc.
Start with a more general format instead to improve our
guessing odds.
* Support content type Structured Name Syntax Suffixes (+json +xml)
If we can't find a fully specified content type in our lookup,
fall back to just the suffix (which is registered with a leading +)
Fixes#2800Fixes#2805
* Harden reconciliation - Fixes#2590
- check for non-JSON / unparseable JSON returns
- handle malformed results response with no name for candidates
- catch any Exception, not just IOExceptions
- call processManager.onFailedProcess() for cleanup on error
* Add default constructor for Jackson
Jackson complains about needing a default constructor for the
NON_DEFAULT annotation, but I'm not sure why this worked before.
* Clean up indentation and unused variable - no functional changes
Make indentation consistent throughout the module, changing recently
added lines to use the standard all spaces convention.
Remove unused count variable
* Simplify control flow
* Update limit parameter comment. No functional change.
* Replace ternary expression which is causing NPE
* Add reconciliation tests using mock HTTP server
* Fixes#486. Builds on code from Steffen Stundzig
- Switch from ICU4J to juniversalchardet
(Java port of Mozilla charset detector)
- Replace org.json code with Jackson
- Add tests
- Add TODO for multi-file character encoding mismatches
* Restore dependency lost in rebase
Co-authored-by: Steffen Stundzig <git@stundzig.de>
* Preserve international characters on import/export
Fixes#1352. Preserve non-ASCII characters in project names on
project creation and filenames on export.
Uses existing filename cleaner with the addition of a few
more characters from StackOverflow, plus "#" which messes
up the download URL. Also URIencode download URL.
Removes unused I18N-incompatible cleaning function from
Wikidata extension rather than fixing it.
* Use common name cleaner function
Also preview cleaned table name instead of raw name, so user can see it.
Also add a TODO for better preview of column names
* Add preferences for the row display quantity
Be able to control the choices for the quantity of rows displayed.
* Added _checkPaginationSize(gridPageSize, defaultGridPageSize)
Added DataTableView._checkPaginationSize(gridPageSize, defaultGridPageSize), gridPageSize = smallest size.
* Update data-table-view.js
Fix missing semi-comma.
* Fix typeof gridPageSize != "object" not working for null
Fix typeof gridPageSize != "object" not working for null
* Update data-table-view.js
* Fix tableHeader instead of headerTable
Fix tableHeader instead of headerTable
* Start for Goto Page link.
First implementation. Is functional.
* Background-color & validation
Background-color for each « button » & validation for the page choice of the user: number, > 1 & < last & singular for a one page project.
* Changed from prompt() to <input type="number">
Changed from prompt() to <input type="number">, and visual X out Y.
* Manage bounds in a sticky way
If the user choose below 1, 1 will be displayed, and if the user choose above the max, the max page will be displayed.
* width of <input> & « of X pages »
Add pages after « of maxValue », calculate the width of <input> based on max value.
* Update data-table-view.js
Little fix.
* Correct min and max for <input>
Correct min and max for <input>
* Managment of the arrow's key down.
Add managment to keep the arrow's key in the CurrentPage <input>.
* Fixes for Thad’s KeyDown's « Infinite Paging »
Fixes for Thad’s KeyDown's « Infinite Paging » & PageSize changes.
* Code rehookCurrentPageInput & spacing
Code rehookCurrentPageInput & spacing for PageSize section
* Update data-table-view.js
Fix & move code in Page Control.
* Wrap in a Try/Catch the currentPageInput.focus()
Wrap in a Try/Catch the currentPageInput.focus() and add a verification of focus change with if(!(currentPageInput.is(":focus"))) window.setTimeout(rehookCurrentPageInput, rehookDelay).
* Fixed currentPageInput != document.activeElement
Fixed currentPageInput != document.activeElement
that was before: !(currentPageInput.is(":focus"))
* Better handling of the « Infinite Paging »
Better handling of the « Infinite Paging » with window.setTimeout(rehookCurrentPageInput, rehookDelay)
* $.i18n('core-views/goto-page', …
$.i18n('core-views/goto-page', '<span id="currentPageInput" />', '<span id="lastPageSpan" />'))
* Fix i18n plural
Fix i18n plural, FR still not fixed.
* Remove {{plural:$2|page|pages}} in french
Remove {{plural:$2|page|pages}} in French, not working.
* Update data-table-view.js
Forgot a semi-comma.
* First implementation
First implementation, is functional.
* Remove setTimeout(refocusCurrentPageInput, refocusDelay);
Remove setTimeout(refocusCurrentPageInput, refocusDelay);, instead, onready after creating the page input field.
* Typo: missing semi-colon
Typo: missing semi-colon
* Fix bug related to stealing focus of facets & added a delay
Fix bug related to stealing focus of facets & added a delay (1 s.) before changing pages.
* Revert "Remove {{plural:$2|page|pages}} in french"
This reverts commit 7274a21ee04e6480e50de39f1a7788164dd1981a.
* Start for Goto Page link.
First implementation. Is functional.
* Background-color & validation
Background-color for each « button » & validation for the page choice of the user: number, > 1 & < last & singular for a one page project.
* Changed from prompt() to <input type="number">
Changed from prompt() to <input type="number">, and visual X out Y.
* Manage bounds in a sticky way
If the user choose below 1, 1 will be displayed, and if the user choose above the max, the max page will be displayed.
* width of <input> & « of X pages »
Add pages after « of maxValue », calculate the width of <input> based on max value.
* Update data-table-view.js
Little fix.
* Correct min and max for <input>
Correct min and max for <input>
* Managment of the arrow's key down.
Add managment to keep the arrow's key in the CurrentPage <input>.
* Fixes for Thad’s KeyDown's « Infinite Paging »
Fixes for Thad’s KeyDown's « Infinite Paging » & PageSize changes.
* Code rehookCurrentPageInput & spacing
Code rehookCurrentPageInput & spacing for PageSize section
* Update data-table-view.js
Fix & move code in Page Control.
* Wrap in a Try/Catch the currentPageInput.focus()
Wrap in a Try/Catch the currentPageInput.focus() and add a verification of focus change with if(!(currentPageInput.is(":focus"))) window.setTimeout(rehookCurrentPageInput, rehookDelay).
* Fixed currentPageInput != document.activeElement
Fixed currentPageInput != document.activeElement
that was before: !(currentPageInput.is(":focus"))
* Better handling of the « Infinite Paging »
Better handling of the « Infinite Paging » with window.setTimeout(rehookCurrentPageInput, rehookDelay)
* $.i18n('core-views/goto-page', …
$.i18n('core-views/goto-page', '<span id="currentPageInput" />', '<span id="lastPageSpan" />'))
* Bump rhino from 1.7.10 to 1.7.12
Bumps [rhino](https://github.com/mozilla/rhino) from 1.7.10 to 1.7.12.
- [Release notes](https://github.com/mozilla/rhino/releases)
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/mozilla/rhino/commits)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* Fix i18n plural
Fix i18n plural, FR still not fixed.
* Remove {{plural:$2|page|pages}} in french
Remove {{plural:$2|page|pages}} in French, not working.
* Update data-table-view.js
Forgot a semi-comma.
* Bump guava from 19.0 to 23.0
Bumps [guava](https://github.com/google/guava) from 19.0 to 23.0.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/compare/v19.0...v23.0)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* Bump testng from 6.9.10 to 7.1.0
Bumps [testng](https://github.com/cbeust/testng) from 6.9.10 to 7.1.0.
- [Release notes](https://github.com/cbeust/testng/releases)
- [Changelog](https://github.com/cbeust/testng/blob/master/CHANGES.txt)
- [Commits](https://github.com/cbeust/testng/commits)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* Bump to Guava 23.6.1-jre
* Bump jasypt from 1.9.2 to 1.9.3
Bumps [jasypt](https://github.com/jasypt/jasypt) from 1.9.2 to 1.9.3.
- [Release notes](https://github.com/jasypt/jasypt/releases)
- [Commits](https://github.com/jasypt/jasypt/commits/jasypt-1.9.3)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* Added conflicts-with constraints (#2641)
Implemented conflicts-with scrutinizer as part of #2354
* Fix a bug introduced by I #1038, PR #2616 (#2684)
Fix a bug introduced by I #1038, where the first tab, in the Wikidata mode, would have a bad background. PR #2616
* Fixed typos made in PR #2641
* Remove setTimeout(refocusCurrentPageInput, refocusDelay);
Remove setTimeout(refocusCurrentPageInput, refocusDelay);, instead, onready after creating the page input field.
* Trim input url whitespace (#2534)
Fixes#2516
* Bump google-http-client-jackson2 from 1.20.0 to 1.35.0
Bumps [google-http-client-jackson2](https://github.com/googleapis/google-http-java-client) from 1.20.0 to 1.35.0.
- [Release notes](https://github.com/googleapis/google-http-java-client/releases)
- [Changelog](https://github.com/googleapis/google-http-java-client/blob/master/CHANGELOG.md)
- [Commits](https://github.com/googleapis/google-http-java-client/compare/1.20.0...v1.35.0)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* Switch to new rhino-runtime component
* Typo: missing semi-colon
Typo: missing semi-colon
* Replace Apache Ant with Commons Compress (#2691)
NOTE: Changes the public API where some of the old types were
embedded which means that any extensions that extend these
interfaces will have to be updated.
Fixes#2690.
* Fix bug related to stealing focus of facets & added a delay
Fix bug related to stealing focus of facets & added a delay (1 s.) before changing pages.
* Revert "Remove {{plural:$2|page|pages}} in french"
This reverts commit 7274a21ee04e6480e50de39f1a7788164dd1981a.
* Update to latest Jython version 2.7.2
Closes#2642
* Bump git-commit-id-plugin from 2.2.4 to 4.0.0
Bumps [git-commit-id-plugin](https://github.com/git-commit-id/maven-git-commit-id-plugin) from 2.2.4 to 4.0.0.
- [Release notes](https://github.com/git-commit-id/maven-git-commit-id-plugin/releases)
- [Commits](https://github.com/git-commit-id/maven-git-commit-id-plugin/compare/v2.2.4...v4.0.0)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* Bump maven-assembly-plugin from 3.1.0 to 3.3.0
Bumps [maven-assembly-plugin](https://github.com/apache/maven-assembly-plugin) from 3.1.0 to 3.3.0.
- [Release notes](https://github.com/apache/maven-assembly-plugin/releases)
- [Commits](https://github.com/apache/maven-assembly-plugin/compare/maven-assembly-plugin-3.1.0...maven-assembly-plugin-3.3.0)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* Bump httpclient from 4.5.5 to 4.5.12
Bumps httpclient from 4.5.5 to 4.5.12.
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* Bump powermock.version from 2.0.2 to 2.0.7
Bumps `powermock.version` from 2.0.2 to 2.0.7.
Updates `powermock-module-testng` from 2.0.2 to 2.0.7
- [Release notes](https://github.com/powermock/powermock/releases)
- [Changelog](https://github.com/powermock/powermock/blob/release/2.x/docs/changelog.txt)
- [Commits](https://github.com/powermock/powermock/compare/powermock-2.0.2...powermock-2.0.7)
Updates `powermock-api-mockito2` from 2.0.2 to 2.0.7
- [Release notes](https://github.com/powermock/powermock/releases)
- [Changelog](https://github.com/powermock/powermock/blob/release/2.x/docs/changelog.txt)
- [Commits](https://github.com/powermock/powermock/compare/powermock-2.0.2...powermock-2.0.7)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* Bump butterfly from 1.0.2 to 1.0.3
Bumps [butterfly](https://github.com/OpenRefine/simile-butterfly) from 1.0.2 to 1.0.3.
- [Release notes](https://github.com/OpenRefine/simile-butterfly/releases)
- [Commits](https://github.com/OpenRefine/simile-butterfly/commits)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* Bump slf4j-api from 1.7.18 to 1.7.30
Bumps [slf4j-api](https://github.com/qos-ch/slf4j) from 1.7.18 to 1.7.30.
- [Release notes](https://github.com/qos-ch/slf4j/releases)
- [Commits](https://github.com/qos-ch/slf4j/compare/v_1.7.18...v_1.7.30)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* Bump commons-validator from 1.5.1 to 1.6
Bumps commons-validator from 1.5.1 to 1.6.
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
* Added translation using Weblate (Portuguese (Brazil))
* Translated using Weblate (Portuguese (Brazil))
Currently translated at 100.0% (47 of 47 strings)
Translation: OpenRefine/gdata
Translate-URL: https://hosted.weblate.org/projects/openrefine/gdata/pt_BR/
* Translated using Weblate (Japanese)
Currently translated at 100.0% (179 of 179 strings)
Translation: OpenRefine/wikidata
Translate-URL: https://hosted.weblate.org/projects/openrefine/wikidata/ja/
* Disables FacetContainer while computing clusters
fixes#2675
* Added translation using Weblate (Portuguese (Brazil))
* Translated using Weblate (Portuguese (Brazil))
Currently translated at 100.0% (740 of 740 strings)
Translation: OpenRefine/Translations
Translate-URL: https://hosted.weblate.org/projects/openrefine/translations/pt_BR/
* Added translation using Weblate (Portuguese (Brazil))
* Translated using Weblate (Portuguese (Brazil))
Currently translated at 100.0% (60 of 60 strings)
Translation: OpenRefine/database
Translate-URL: https://hosted.weblate.org/projects/openrefine/database/pt_BR/
* Translated using Weblate (Portuguese (Brazil))
Currently translated at 6.7% (12 of 179 strings)
Translation: OpenRefine/wikidata
Translate-URL: https://hosted.weblate.org/projects/openrefine/wikidata/pt_BR/
* Translated using Weblate (Japanese)
Currently translated at 100.0% (60 of 60 strings)
Translation: OpenRefine/database
Translate-URL: https://hosted.weblate.org/projects/openrefine/database/ja/
* Translated using Weblate (Japanese)
Currently translated at 100.0% (740 of 740 strings)
Translation: OpenRefine/Translations
Translate-URL: https://hosted.weblate.org/projects/openrefine/translations/ja/
* Translated using Weblate (Japanese)
Currently translated at 100.0% (60 of 60 strings)
Translation: OpenRefine/database
Translate-URL: https://hosted.weblate.org/projects/openrefine/database/ja/
* Added translation using Weblate (Bengali (India))
* Load GDrive icon from local resource, not Github (#2689)
Fixes#2688.
* Convert illegal characters into legal ones. (#2431)
* Convert illegal characters into leagal ones.
* Test tab in key & value string
Also fix up test that depended on previous TAB
related error message and clean up logging
Co-authored-by: Tom Morris <tfmorris@gmail.com>
* Bump signpost-commonshttp4 from 1.2.1.2 to 2.0.0 (#2695)
Bumps [signpost-commonshttp4](https://github.com/mttkay/signpost) from 1.2.1.2 to 2.0.0.
- [Release notes](https://github.com/mttkay/signpost/releases)
- [Changelog](https://github.com/mttkay/signpost/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mttkay/signpost/compare/1.2.1.2...oauth-signpost-2.0.0)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
* Add comment to help designers understand TreeData (#2715)
* Update jquery.i18n to 1.07 and fix non-English plural support (#2717)
* Update jquery.i18n to 1.07 and add missing rule parser
Fixes#2700
Adds missing CLDRPluralRuleParser.js so that plurals are supported.
Updates all files to jquery.i18n 1.07
Includes a bunch of specialty language support,
but only Finnish and Russian are loaded as examples.
* Add some missing translations, including plurals
Fix some cases of Javascript string concatenation and
plural conditionalization to demonstrate that plurals work
in both English and French now.
NOTE: Corresponding updates need to be made to all the other
language files since some keys were renamed or eliminated.
* Unused imports and other minor cleanups (#2723)
* Two minor fixes
- prevent invalid index error on empty strings (shouldn't normally happen)
- update deprecated Apache Commons Lang method
* Remove unused imports
* Remove feature Edit Facet Name
Remove feature Edit Facet Name that got merged by mistake.
* Remove feature Edit Facet Name
Remove feature Edit Facet Name that got merged by mistake.
* Spacing
Spacing
* Fix the delay, adjust to .2 s.
Fix the delay, adjust to .2 s. and stop using promises.
* Fix headerTable to tableHeader
Fix headerTable to tableHeader (PR #2719)
* data-header-table to data-table-header
data-header-table to data-table-header
* Remove .data-header-table-container
Remove .data-header-table-container
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Tom Morris <tfmorris@gmail.com>
Co-authored-by: Ekta Mishra <ektamishra1999@gmail.com>
Co-authored-by: Nishtha <51858166+Nishtha3512@users.noreply.github.com>
Co-authored-by: Thad Guidry <thadguidry@gmail.com>
Co-authored-by: Rafael Fontenelle <rafaelff@gnome.org>
Co-authored-by: Isao Matsunami <isao.matsunami@gmail.com>
Co-authored-by: Biswaranjan Manna <manna.biswaranjan@gmail.com>
Co-authored-by: chuhao zeng <32441682+zengchu2@users.noreply.github.com>
* Use ContentDisposition instead of ContentType to control download
Fixes#1197. Previously we were using a funky ContentType to attempt
to force a file download rather than display in browser, but this
conflicted with attempts to save UTF-8 which was outside the Basic
Multilingual Plane (BMP).
By switching to ContentDisposition: attachment, which has been
the preferred method for a number of years, we can avoid this conflict.
As part of this, switch to using the "preview" param consistently
to control preview vs download rather than the content type.
* Switch content type to text/plain
Now that we don't need to use ContentType to control download
behavior, we can use something more reasonable.
* feat: Bounding the table to a single container
* Making the table more responsive
* fixing design
* remove references to adjustDataTables
* remove useless table-scroll function
* Fix naming and restore old styling
* fix border top & font size
* fix firefox css bug
* fix for XML/JSON Importers
* Fixed styling
* fixed typos
* fixed column rendering and added a sticky header
* fixed firefox borders
* fixed border and table background
* fix importer stylings (td)
Co-authored-by: kushthedude <kushthedude@gmail.com>
* Use mockwebserver instead of live network for tests
Fixes#2680. Fixes#1904.
* Remove use of deprecated methods
* Convert to use Apache HTTP Components client library
Fixes#1410 by virtue of redirect following being a built-in
capability of the library, along with retries with binary backoff,
built-in decompression, etc.
* Address review comments
Also add explicit dependency on Apache commons-io
which should be a transitive dependency, but doesn't
work for Jena 3.8+
Tried switching to standard packaging of
apache-jena-libs, but that doesn't know how
to find the JSONLD parser.
* Fix bug in choice counts for records mode
* Add test for value grouper on records
* Refactor and comment code
* Count distinct instances of null/blank data
* Update test to check for blank data count in records
* Remove unnecessary import statement
* added options ui
* added definition for both separators
* added tests
* removed definitions from backend and added them to frontend
* added reverse order and handling for accented characters
* added tests for accented characters and reverse split
* fixed build errors
* unicode character ranges instead
* added examples
* Update jquery.i18n to 1.07 and add missing rule parser
Fixes#2700
Adds missing CLDRPluralRuleParser.js so that plurals are supported.
Updates all files to jquery.i18n 1.07
Includes a bunch of specialty language support,
but only Finnish and Russian are loaded as examples.
* Add some missing translations, including plurals
Fix some cases of Javascript string concatenation and
plural conditionalization to demonstrate that plurals work
in both English and French now.
NOTE: Corresponding updates need to be made to all the other
language files since some keys were renamed or eliminated.
* Convert illegal characters into leagal ones.
* Test tab in key & value string
Also fix up test that depended on previous TAB
related error message and clean up logging
Co-authored-by: Tom Morris <tfmorris@gmail.com>
