* Templating exporter should have clear error reporting
* Using i18n for meesage and removed the unnecessary null check
* Removed usage of bindings
* Adding tests for grel, renamed ParsetTests to TemplatingParserTests
* Regex to test the keys of the template exporter
* Cancel changes to the templating parser
Co-authored-by: Antonin Delpeuch <antonin@delpeuch.eu>
This pull request comes without tetsts because there is no clean way
to set the system timezone in a Java unit test:
https://stackoverflow.com/questions/23466218/setting-timezone-for-maven-unit-tests-on-java-8
The problem came from the fact that we were interpreting all dates with the current
zone *offset*, which varies during the year depending on DST, meaning that during the winter
we would interpret summer dates in a winter timezone, which does not make sense.
This changes the date conversion mechanism to only rely on the current zone, making sure
the correct offset is used depending on the value of the converted date.
* Change the behaviour of failed reconciliations to not reconcile the cell at all.
This makes it easier to tell apart cells which could not be reconciled due to
a network error and those which just did not have any reconciliation candidate.
This makes it possible to retry reconciling cells which have been left unreconciled
after a recon operation.
Closes#3369.
* Update StandardReconConfigTests with new behaviour for failed recons
By default, the Apache HTTP client does not retry POST requests in those cases
because they are not deemed idempotent. But for reconciliation they are, so they
should be retried.
Closes#3664.
Null values in non-null cells should normally not happen, but sometimes they do and
we shouldn't fail miserably in those cases.
Closes#2880.
Co-authored-by: Tom Morris <tfmorris@gmail.com>
* Update test to demonstrate retry mechanism for recon queries.
This was fixed earlier by #3237. Closes#3369.
* Ensure non-zero retry interval in HttpClient
* Restore original bound
Improvements to GREL Help text in-tool and minor updates to GREL reference. Plus removal of pointless tests
Co-authored-by: Antonin Delpeuch <antonin@delpeuch.eu>
Co-authored-by: Owen Stephens <owen@ostephens.com>
* Refactor HTTP code into a common utility class
Centralizes the six (slightly) different implementations to use
a common Apache HTTP Client 5 implementation which implements our
strategies for retries, timeouts, error handling, etc.
Apache HTTP Client 5 adds support for Retry-After headers, HTTP/2,
and a bunch of other stuff under the covers.
Moves request delay to a request interceptor and fixes calculation
of the delay (again). Increase retries from 1x to 3x and use delay*2
as the default retry interval, if no Retry-After header. Uses an
exponential backoff strategy for multiple retries.
* Reuses HTTP client across requests
* Use IOException instead of Exception for HTTP errors
* Fix: Preventing addition of any empty cells with whitespaces while importing Xml data with Tests : Issue #1095
* Chore: Using 'CharMatcher' to match whitespace pattern instead of using custom regex : Issue #1095
* add clarity for reinterpret docs
Helps fix#3292
* update reinterpret docs phrasing
We agreed to use "encoding" to be friendly to user exposed messaging instead of "encoder" and "decoder" that is used internally.
* fix serializeReinterpret() test json
* changed to reflect the function's acceptance of either simple string or regex
* cast p into a Pattern
* cast p into a Pattern
* Changed test to reflect the new output from function.
* Add more keyer tests
- All forms of Unicode whitespace for both fingerprint & N-gram fingerprint
- additional N-gram fingerprint cases
* Improve fingerprint keyers
- Update N-gram fingerprint keyer to match (missed last time)
- refactor string normalization to reduce redundancy between two keyers
- add C1 controls to control characters that are stripped
- include all Unicode whitespace characters in splitting delimiter
and don't strip controls which are whitespace (HT, LF, VT, FF, CR,
NEL)
- minor cleanups, simplifications, and performance optimizations
* Visually center links box
* Add the Java runtime info to the About page - fixes#3240
- Add the Java runtime name & version GetVersionCommand
- Add the returned information to the About page
* Clean up importer refactoring
Remove an extra copy of filename setting.
Revert some additional API changes (retaining both versions)
* Revert archive file name changes & mark as deprecated
* Add utility helpers to create array of comparable items
* Extend sort() to handle arrays with nulls
- Instead of NullPointerException on nulls, sort them last
- add JSON helpers to return Comparable[] in addition to Object[]
- Non-homogenous arrays or arrays with non-primitive
objects (array or object) are not sortable
- Add tests for both new and old sort functionality
* Refactor GREL Get tests
- move helper up to RefineTest
- move tests to the correct module
* Extend forEach() to support JSON objects - fixes#3149
Also add tests for existing forEach forms in addition to the new one
* Add a couple more tests
* Migrate reconciliation calls to OkHTTP, for #2903
* Migrate to Apache HTTP Commons
* Migrate data extension to Apache HTTP client
* Deprecate HttpURLConnection in RefineServlet
* Use LaxRedirectStrategy, clean up imports
* Remove read and pool timeouts, only keep the connection timeout
* Adapt mocking of HTTP calls after migration
* Fix ToDate test failure - fixes#3026
Instead of computing offset from UTC at current
point in time, use the offset from the parsed
date so that we're not affected by crossing
a daylight savings time boundary.
* Fix date parsing with locale as first format string
Also refactors for simpicity, restore some dropped tests,
and restores previous behavior of considering a bad
format string an error instead of silently ignoring it.
It does NOT address another issue which was introduced
in May 2018 of treating date/times without timzone
information as UTC instead of local.
* Restore error checking and messages
* Save & restore default timezone for tests
Also add some ToDos for places where LocalDate is being misused.
* Make sure data directory is directory, not a file
* Add a test for zip archive import
Also tests the saving of the archive file name and source filename
* Add TODOs - no functional changes
* Cosmetic cleanups
* Revert importer API changes for archive file name parameter
Fixes#2963
- restore binary compatibility to the API
- hoist the handling of both fileSource and archiveFileName from
TabularImportingParserBase and TreeImportingParserBase to
ImportingParserBase so that there's only one copy. These 3 classes are
all part of the internal implementation, so there should be no
compatibility issue.
* Revert weird flow of control for import options metadata
This reverts the very convoluted control flow that was introduced
when adding the input options to the project metadata. Instead
the metadata is all handled in the importer framework rather than
having to change APIs are have individual importers worry about
it.
The feature never had test coverage, so that is still to be added.
* Add test for import options in project metadata & fix bug
Fixes bug where same options object was being reused and overwritten,
so all copies in the list ended up the same.
* Fix text guesser so it doesn't guess wikitext
Fixes#2850
- Add simple magic detector for zip & gzip files to keep
it from attempting to guess binary files
- Add a counter for C0 controls for the same reason
- Tighten wikitable counters to require marker at
beginning of the line, per the specification
- Refactor to use Apache Commons instead of private
counting methods
- Add tests for most TextGuesser formats
* Remove misplaced duplicate test data file
* Fix LGTM warning + minor cleanups
* Use BoundedInputStream to prevent runaway lines
* Add utility functions to check/convert dates
* Add date tests and refactor to DRY up
* Fix date import - fixes#1908
Change from java.util.Date to OpenRefine 3.0+'s OffsetDateTime
Fixes#1908
* Centralize date conversion
* Moving utility methods to ParsingUtilities
* Fix tests
* Use standard text normalization - fixes#2898Fixes#2898. Fixes#409. Refs #650
Replaces homegrown ISO Latin-1 only character subsitition
with standard Java Normalize to NFD, followed by diacritic
removal and a few custom character expansions/replacements.
* Fix Mac build
* Improve compatibility with previous code
One intentional change is folding O with stroke to
oe instead of o.
- Use more powerful NFKD instead of NFD
- strip punctuation after decomposition since it can generate
new punctuation
- Add compatibility test for old asciify() method
- Add some graphically similar characters to substitution table
* Add oe character/ligature & more long S forms
* More tests for ligatures and Latin Extended
* Add Latin-1 Supplement tests
