Stefano Mazzocchi
a286b805d5
no reason to process the contextual rows while clustering
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@440 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-10 08:11:31 +00:00
David Huynh
70449cf7c8
Better error catching in toNumber function.
...
Watch out for the string "Infinity" while importing data sets: don't parse it into a double.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@438 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-09 21:59:50 +00:00
David Huynh
8950e87e02
When re-loading existing projects from disk, cache recon objects by their IDs to lower memory consumption.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@437 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-09 19:32:50 +00:00
David Huynh
dff870519e
Made numeric range index handle value.log() properly when value is 0 or negative.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@436 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-09 05:24:46 +00:00
David Huynh
a0d8c385f9
Do a bit more checking when retrieving project metadata just in case project metadata is null.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@435 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-09 04:52:32 +00:00
Stefano Mazzocchi
d3d40d608a
bunch of PMD-induced fixes
...
(now the PMD report is clean)
git-svn-id: http://google-refine.googlecode.com/svn/trunk@430 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-09 00:14:11 +00:00
David Huynh
759824e1b4
Bug fix: editing one facet choice while some other choices are selected resulted in no change.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@429 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-08 22:16:12 +00:00
Stefano Mazzocchi
7526c4e582
cleanups (no functional changes)
...
this makes pmd and javac on linux happier
git-svn-id: http://google-refine.googlecode.com/svn/trunk@427 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-08 20:46:02 +00:00
Will Moffat
efb9a43b73
Added: toDate() now supports formats.
...
My dataset has the same dates written in two styles: Nov-09 and 11/09
You can now parse this using: value.toDate2('MM/yy','MMM-yy').toString('yyyy-MM')
git-svn-id: http://google-refine.googlecode.com/svn/trunk@414 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-07 19:12:08 +00:00
David Huynh
0996b9e1dd
Gzip project export tar files.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@394 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-06 22:28:30 +00:00
David Huynh
5320cc6587
Make duplicated column names unique during import by appending indices to them.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@392 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-06 17:55:36 +00:00
David Huynh
9d9329ca96
Implemented row remove command.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@391 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-06 07:47:44 +00:00
David Huynh
1fd85c62bf
Implemented column rename command.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@390 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-06 07:15:34 +00:00
David Huynh
93d6f9fc54
Better error message for bad regular expressions in GEL.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@388 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-06 06:18:07 +00:00
David Huynh
f402db10af
Implemented inter-project joins.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@387 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-06 05:35:48 +00:00
Stefano Mazzocchi
771810bc0d
avoid exception if there is only one extension in the whole archive
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@385 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-05 21:36:27 +00:00
Stefano Mazzocchi
2efbf0031f
- removed the 'thirdparty' directory (now the 'gridworks' script will download and install needed tools if they are not present in the system already)
...
- added 'findbugs' command that uses the findbugs static analyzer to look for problems in the code
- fixed a bunch of issues that findbugs found (a few methods would go a little faster, and a few NPE will be avoided... nothing major but good to have)
git-svn-id: http://google-refine.googlecode.com/svn/trunk@382 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-05 07:15:16 +00:00
Stefano Mazzocchi
798b2a36ca
- archive and compressed file importer (supports zip, tar, gz, bz2, tar.gz and tar.bz2)
...
(works by loading the files that have the most common extensions in the archive)
- changed default max heap to 3Gb
git-svn-id: http://google-refine.googlecode.com/svn/trunk@381 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-04 07:48:47 +00:00
Stefano Mazzocchi
c24ec94835
had to shuffle around a bunch of classes to separate the main server classloader from the context classloader and allow reloading to happen for real
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@377 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-03 01:01:11 +00:00
Stefano Mazzocchi
72203cd3d5
- moved all code that contained MIT IP outside ( http://code.google.com/p/simile-vicino/ )
...
- moved bzip2 and tar code from apache ant into their own jar files
- now gridworks source contains only com.metaweb.* code everything else is a jar dependency
- started to work on archive importer
git-svn-id: http://google-refine.googlecode.com/svn/trunk@376 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-02 23:40:12 +00:00
Stefano Mazzocchi
4eda7ae2c0
avoid an array out of bounds exception in case there are no columns in the dataset
...
(I know, it should not happen but when it does let's not barf)
git-svn-id: http://google-refine.googlecode.com/svn/trunk@375 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-02 20:21:41 +00:00
Stefano Mazzocchi
62f5f21ca3
atom is handled as well by the XML importer
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@374 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-02 06:44:05 +00:00
Stefano Mazzocchi
83faee3aa9
add a frame-less menu item in macosx to be able to open another gridworks browser window/tab in case we closed it by mistake
...
(no idea how to do this on windows, though, since there is no frame-less menu concept there)
git-svn-id: http://google-refine.googlecode.com/svn/trunk@373 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-02 06:39:41 +00:00
Stefano Mazzocchi
521acda025
- pass the svn revision as format version (for more detailed verification)
...
- add an 'autoreload' setting that makes Gridworks autoreload its self if a class gets changed
(this is useful to make development cycles faster when working on the java code with autocompiling IDE like Eclipse or IDEA)
git-svn-id: http://google-refine.googlecode.com/svn/trunk@372 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-02 00:52:38 +00:00
Stefano Mazzocchi
988378c761
Hmm, String.split() bites us again: use the commons-lang one instead to avoid having to escape regexp values (this was preventing a user from splitting by "." in GEL)
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@370 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-01 17:49:31 +00:00
Stefano Mazzocchi
0e07ec7acc
crude, I know, but for now make Gridworks digest RDF/XML as it was XML (works surprisingly well, btw)
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@369 7d457c2a-affb-35e4-300a-418c747d4874
2010-04-01 16:56:38 +00:00
Stefano Mazzocchi
dced641599
- added the ability to specify the character separator for CSV or TSV files that don't use commas or tabs (this was needed to parse a dataset that we got from the BBC to try things out)
...
- used commons-lang split function instead of the java String.split one, this is necessary to avoid having to escape separators that might be confused for regexps
git-svn-id: http://google-refine.googlecode.com/svn/trunk@368 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-31 22:34:21 +00:00
Stefano Mazzocchi
7c132cfa53
clean eclipse warnings
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@357 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-29 20:51:18 +00:00
David Huynh
1d0e6abaf8
Got some work done on the plane:
...
- better detection of record XML elements in XML importer
- XML importer creates column groups and data table view renders them
git-svn-id: http://google-refine.googlecode.com/svn/trunk@356 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-27 05:23:09 +00:00
David Huynh
2a9fbd7d81
Made sure columns are named hierarchically in XML importer.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@355 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-24 23:15:10 +00:00
David Huynh
df7389876f
First shot at XML import.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@354 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-24 23:08:08 +00:00
David Huynh
4e76155652
Use application/x-unknown when exporting TSV so that the browser just saves the result. This is good for large exports, which overload the browser if the browser tries to display them.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@353 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-24 17:51:27 +00:00
David Huynh
47cad64a3f
Properly unescape \t, \r, \n, \\.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@352 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-24 17:50:44 +00:00
David Huynh
30e3ca4965
Added splitByLengths function.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@351 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-24 04:30:31 +00:00
David Huynh
4df1c4107a
Fixed a bug introduced recently: recon candidates were not serializing their topic types for the data view, so in the data view we can't send back a candidate's types when the user wants to match the candidate to some cells. I need to figure out a better way to optimize this.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@350 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-24 03:58:52 +00:00
David Huynh
85d1671d6e
Fixed minor bug: recon wasn't saving out its candidates if its judgment is Matched. So when a project is saved and reloaded, it loses all of the recon candidates except for the matches.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@344 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-23 18:02:08 +00:00
David Huynh
2846d66261
Detect max cell index on load, just in case the max cell index we've stored previously was out of whack.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@341 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-23 03:19:17 +00:00
David Huynh
f8d30e9e8e
Don't send back recon candidate types for rendering cells.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@340 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-23 01:17:45 +00:00
David Huynh
c07ba83a36
Don't send back recon candidate types for rendering cells.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@339 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-23 01:17:32 +00:00
David Huynh
3dc4db020f
Support quick undo of the last operation (Ctrl-Z).
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@338 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-23 00:26:28 +00:00
David Huynh
f5d270e35a
Fixed "off by one bucket" bug in range facet's binning algorithm.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@336 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-22 22:04:29 +00:00
David Huynh
1d20b33cf1
Documented the history package.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@334 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-22 18:19:31 +00:00
David Huynh
ec0110d65b
Documented gel.* packages.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@333 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-22 05:44:35 +00:00
David Huynh
60dd7eab82
Documented expr.* packages.
...
Converted some tabs into spaces.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@332 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-22 00:54:56 +00:00
David Huynh
60f60507f7
Fixed minor bug introduced recently into the Export Project menu command.
...
Documented the commands.* packages.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@331 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-21 20:26:35 +00:00
David Huynh
7648126a5e
Made facets deal with java.util.Collection rather than just Object[].
...
Documented the browsing.* packages.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@330 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-21 07:14:39 +00:00
David Huynh
d90e75dff1
Started a round of documentation.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@329 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-20 23:56:28 +00:00
David Huynh
a43b2a72c1
Made various GEL functions and the forEach control work with java.util.List and java.util.Collection in addition to just Object[].
...
Added field columnNames to row object.
Added 1-bounded numeric log facet.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@328 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-19 23:04:17 +00:00
Vishal Talwar
6fba7d1e7f
sped up jython evaluation by calling function directly instead of invoking parser on string representation of function call
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@327 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-19 21:45:01 +00:00
David Huynh
ff0049307e
Increased file upload size limit to 1GB.
...
Fixed charset detector to be more robust in trying more than one charset.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@326 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-19 19:53:11 +00:00
David Huynh
8dd0dea472
Try to roundtrip reconciled IDs as much as possible when import/export as Excel files.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@322 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-19 00:32:52 +00:00
David Huynh
07cf85b2a5
Added make_cli option for generating a zip containing all files necessary to do 'gridworks run' at the command line. This excludes Java source files but contains pretty much everything else.
...
Added make_all option that makes dmg, exe, and cli.
Added html and xls exporters.
Made exported files named after project names rather than project IDs.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@314 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-17 22:30:16 +00:00
David Huynh
07945f9cde
A more helpful error message when the excel importer fails.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@313 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-17 21:43:58 +00:00
David Huynh
cd062cf028
Minor bug: recon candidate's "id" field should return id, not name.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@312 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-17 19:54:27 +00:00
David Huynh
b26160dc2b
Hopefully a more robust way to get the user data dir on Windows, especially on Windows Vista 64-bit, which jdatapath.dll isn't built for.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@311 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-17 03:51:58 +00:00
David Huynh
1a8a236cdd
Added an error page for when a project create operation fails.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@307 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-16 18:40:07 +00:00
David Huynh
084a6114d7
Track freebase types of columns added with data from Freebase, so that we can later add more data based on those columns. Fixed minor bug in serialization of data extension records.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@303 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-16 06:18:00 +00:00
David Huynh
c6e7986206
Extend data operation is working.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@301 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-16 00:24:20 +00:00
David Huynh
c30a5126df
More work on the extend data preview dialog: columns can now be removed.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@299 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-15 19:44:33 +00:00
David Huynh
a32273de70
More work on the extend data preview dialog. Results now looks correct, but we still are not handling CVTs.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@295 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-14 20:55:57 +00:00
David Huynh
e35c4c3b94
Minor bug.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@294 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-14 05:15:23 +00:00
Stefano Mazzocchi
f7ab7c9cf6
- incorporated Paolo Ciccarese's fixes for VPTrees in Vicino
...
- moved all clustering stuff in the vicino package space to simplify external collaboration on that code
- added "type" function to the GEL
git-svn-id: http://google-refine.googlecode.com/svn/trunk@292 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-13 09:34:17 +00:00
David Huynh
99ae7dea29
More work on the extend data preview dialog. It's starting to render some results.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@290 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-13 08:08:25 +00:00
David Huynh
f34577ec85
Improved grid layout CSS rules.
...
Started working on extending data from Freebase feature.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@289 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-13 07:13:18 +00:00
David Huynh
67bac099f0
- Made removing facet a bit more interactive: the removed facet disappears right away.
...
- Made list facets limit themselves to only 2000 choices, so not to overload the browser.
- Made list and range facets handle errors in expressions better.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@287 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-12 20:08:50 +00:00
Stefano Mazzocchi
00a81c5fc4
make the kNN clustering report the right counts for the facet values (and order them in the clusters by counts)
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@286 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-12 19:10:22 +00:00
Stefano Mazzocchi
d72c07b715
latest clustering fixes (the vptree is still too slow though, I'll probably abandon that approach for now)
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@285 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-12 07:37:37 +00:00
David Huynh
58450555e9
Allow GEL identifiers to contain underscores.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@284 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-12 06:38:12 +00:00
David Huynh
025eccce4b
Implemented "record" field for each row.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@283 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-12 06:33:03 +00:00
David Huynh
af3cb76056
Added support for including dependent rows in row visiting. Facets still don't count them, though.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@282 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-12 01:06:23 +00:00
David Huynh
7e2667ab45
Minor bug in Excel importer: we forgot to update the max cell index.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@281 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-12 00:23:01 +00:00
David Huynh
e760750b57
Fixed minor bug that prevented column details from getting passed on to recon service.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@280 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-11 21:55:32 +00:00
David Huynh
c3ebb5a9f4
Got Vishal's jython integration to work.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@277 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-11 19:56:43 +00:00
Vishal Talwar
4bb9e06772
adding extremely crude jython and clojure expression evaluation
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@276 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-10 23:43:21 +00:00
David Huynh
86a8e13d88
Added "numeric" choice to numeric range facets.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@272 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-10 08:48:04 +00:00
David Huynh
b1fca11342
Made recon use cells from context rows.
...
Fixed bug in menu left-right positioning.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@271 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-10 08:32:20 +00:00
Stefano Mazzocchi
a7d4951725
several improvements for clustering
...
- added a unicode ASCII-fiying addition to the fingerprinting functions
- removed all distance functions for kNN that didn't seem to do anything useful
- added the ability to indicate what value to use as cluster centroid by simply clicking on it
(this is useful for those names that have non-ASCII chars that might not even be on your keyboard.. and cut/paste is error prone/cumbersome)
- added a 10x multiplier to the PPM compression distance which makes it more aligned with the levenshtein ones
- made sure that we construct a phonetic fingerprint for the whole string and not just the beginning subset
(performance is still not ideal but it's now reasonable)
git-svn-id: http://google-refine.googlecode.com/svn/trunk@268 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-10 07:45:14 +00:00
David Huynh
6bf5418f9d
Cell changes should also flush column precomputes.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@267 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-10 07:42:57 +00:00
David Huynh
e008332399
- make recon changes flush column precomputes
...
- fixed bug where recon features are not saved to file properly
- support selecting non-numeric, blank, and error choices in numeric range facets
git-svn-id: http://google-refine.googlecode.com/svn/trunk@265 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-10 06:02:36 +00:00
Stefano Mazzocchi
72b012971f
more polish on the clustering dialog
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@264 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-10 01:18:41 +00:00
Stefano Mazzocchi
c03b223a78
fixed another NPE bug
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@262 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-09 22:43:49 +00:00
David Huynh
51b38a4eed
Fixed minor bug in binning clusterer.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@261 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-09 21:28:40 +00:00
Stefano Mazzocchi
0c4b79c53a
more polish on the clustering dialog
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@260 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-09 20:58:51 +00:00
David Huynh
6b3a20dc46
Check for null confidence string.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@259 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-09 20:57:46 +00:00
David Huynh
2bac6844e2
Fixed csv importer to handle escaped quotation marks ("").
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@257 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-09 19:10:55 +00:00
Stefano Mazzocchi
8ce21461cb
getting closer to the desired functionality... still way too slow though
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@256 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-09 17:28:50 +00:00
Stefano Mazzocchi
50e58fb863
ngram-blocking gives more expected results... but slow as hell, maybe bug in the vptree code?
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@255 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-09 09:05:20 +00:00
Stefano Mazzocchi
546f87a536
let's try with another knn method
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@254 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-09 08:09:35 +00:00
David Huynh
311d15f493
Re-organized column header popup menus and added a bunch of common facets and common cell edit transforms.
...
Added native syntax for regex in GEL and modified replace, split, partition, and rpartition functions to support regex. Removed function replaceRegex.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@249 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-09 06:57:08 +00:00
Stefano Mazzocchi
5b079b04b7
- moved from float to double to avoid excessive casting from secondstring
...
- added a few of the more powerful distances
- fixed a bug in the VPTree builder (although is still not working as I expect it to)
git-svn-id: http://google-refine.googlecode.com/svn/trunk@248 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-09 05:11:36 +00:00
David Huynh
4a4ae6bf27
Fixed toTitlecase to handle parentheses and other delimiters.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@240 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-08 19:40:51 +00:00
David Huynh
ac50b3c48b
Re-worked the cell editor popup.
...
Don't keep logging "Saved workspace."
git-svn-id: http://google-refine.googlecode.com/svn/trunk@235 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-08 06:30:47 +00:00
David Huynh
5d3a57eeeb
Implemented project import and export commands (from/to .tar files).
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@234 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-08 02:34:25 +00:00
David Huynh
a1ec0ea8df
When saving projects, save only modified ones.
...
Save projects and workspace periodically.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@232 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-08 00:37:06 +00:00
David Huynh
3388c3e09f
Still some old Serializable stuff to remove.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@228 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-07 23:02:57 +00:00
David Huynh
80e6111a92
Added options for omitting error and blank choices in list facets, and use them in the various recon facets.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@227 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-07 22:54:02 +00:00
David Huynh
694f09fb0a
Major refactoring: everything is now saved to disk using our own formats, mostly json-based, some inside zip files.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@226 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-07 22:37:26 +00:00
Stefano Mazzocchi
f7b0caa1b8
now kNN clustering is fully operational... not very practical though, needs more work and testing
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@225 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-07 08:27:13 +00:00
David Huynh
e06d8fe130
Better checking for null value in Cell.load.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@224 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-07 00:35:44 +00:00
David Huynh
e0d72c81e9
Renamed "facet-based edit" operation and command to "mass edit", because it's not just facet-based.
...
Added option "apply to other cells with same original content" to single cell edit popup, so it can be used like a find&replace operation.
Renamed "do-text-transform" operation and command to just "text-transform".
git-svn-id: http://google-refine.googlecode.com/svn/trunk@223 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-07 00:25:00 +00:00
David Huynh
253874b1a1
Got Clusterer to use Column.name rather than Column.headerLabel now.
...
Tried using Verdana instead of Tahoma as the common font.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@220 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-06 22:48:42 +00:00
Stefano Mazzocchi
976c1da5c7
much improved facet clustering dialog and functionality
...
NOTE: kNN clustering code operational but is not working as expected
git-svn-id: http://google-refine.googlecode.com/svn/trunk@219 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-06 10:17:58 +00:00
David Huynh
db824bffeb
Fixed bug in saving recon changes.
...
Fixed bug in discard recon judgment operation.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@218 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-06 08:03:29 +00:00
David Huynh
78b1eb7e73
Major refactoring:
...
- Made all Change classes save to and load from .zip files.
- Changed Column.headerLabel to Column.name.
- Save project's raw data to "raw-data" file for now. We'll make it save to a zip file next.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@217 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-06 07:43:45 +00:00
David Huynh
589b9cd936
Re-organized popup menus for row operations. Added filter row.starred.
...
Disabled rendering of key column and column groups for now.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@216 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-05 22:31:47 +00:00
David Huynh
5c845f06bf
Now we can delete a project even if it hasn't been saved to file yet.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@214 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-05 19:43:08 +00:00
David Huynh
b3ac945c33
Implemented single-cell editing.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@210 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-05 08:11:48 +00:00
David Huynh
40cdf5092b
Better display of Calendar objects in data table view and in expression preview dialog.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@208 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-05 02:25:27 +00:00
David Huynh
87d20f3299
Fixed minor bug in numeric bin index where if a value was infinity, the bin count went negative.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@207 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-05 02:10:24 +00:00
Stefano Mazzocchi
37e37488ec
ability to delete a project from the front page
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@206 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-05 01:52:55 +00:00
David Huynh
1d6db8fa6e
Made recon process cause the client page to create facets when the recon process is done.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@203 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-05 01:13:59 +00:00
Stefano Mazzocchi
32c0bf08c9
adding now() and inc() functions to the gel
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@202 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 20:53:07 +00:00
David Huynh
9d8b746121
Switched Cell.value from Object to Serializable.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@201 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 19:59:31 +00:00
David Huynh
3e0ac50e17
Fixed date parsing bug in index.js introduced since last commit.
...
Removed debugging console.log() call in browsing-engine.js.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@200 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 19:38:23 +00:00
David Huynh
1dc3d4abbd
Save project metadata to disk as JSON now rather than through Java serialization API.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@199 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 19:15:46 +00:00
Stefano Mazzocchi
409b451085
started work on protocol buffers
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@197 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 08:46:33 +00:00
David Huynh
22f226358d
Added pre-canned facets isBlank(value) and isError(value).
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@196 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 07:32:30 +00:00
David Huynh
6811f54f31
Fixed quoting bug in tripleloader transposer.
...
Implemented tripleloader exporter.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@194 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 07:02:03 +00:00
Stefano Mazzocchi
8f01da0aa8
fixing the date parser
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@193 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 04:06:38 +00:00
Stefano Mazzocchi
1695e2f8f1
add the ngramFingerprint function
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@191 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 02:37:25 +00:00
Stefano Mazzocchi
5c3ca7723a
use a TreeSet to do both sorting and de-dupe of the split fragments
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@190 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 02:37:06 +00:00
David Huynh
70df6821a0
Made expression preview dialog for text transform operation also support repeat option.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@189 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 02:25:42 +00:00
Stefano Mazzocchi
cde6a02cbb
typo (needed to escape ' which is actually *not* an HTML entity, who knew)
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@188 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 02:24:06 +00:00
David Huynh
87956be756
Minor bug: don't try to bind null cell value.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@187 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 02:14:40 +00:00
David Huynh
1f05954924
Fixed regex text search facet to handle errors better. Use .text() rather than .html() to render cell values, or & will not show up.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@184 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 01:47:58 +00:00
David Huynh
72d06fe65c
Added support for canceling running and pending processes.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@183 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 01:14:48 +00:00
David Huynh
eaef7b2394
Also let user decide what to do on expression evaluation error when creating a new column.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@182 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 00:32:54 +00:00
David Huynh
5a0a8bea4f
Added custom dialog box for create column operation (with a field for the new column name).
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@180 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 00:12:39 +00:00
David Huynh
2fe8f98e4e
Added repeat and repeatCount options for text transform operation. This lets us fix those & repeated encoding problems easily.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@179 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-04 00:00:46 +00:00
David Huynh
b4d2cef526
Added an option for what to do when a text transform errors out. Made a custom expression preview dialog for the text transform command in order to suppor that option.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@178 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-03 22:12:48 +00:00
David Huynh
c1498448e4
Implemented global and per-project expression histories and hooked them up to the expression preview dialog.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@176 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-03 21:21:38 +00:00
David Huynh
b75f1faea8
Changed tabs to spaces. No functionality change.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@174 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-03 04:19:58 +00:00
Stefano Mazzocchi
2691ee50d7
adding OS-specific data paths
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@173 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-03 02:53:07 +00:00
David Huynh
ad7671508f
Added "cancel processes" command, not hooked up yet.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@171 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-03 00:30:39 +00:00
David Huynh
59c5314e42
Fixed bug in list facet: list facets on columns with numeric data weren't working before.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@169 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-02 23:07:33 +00:00
David Huynh
3ecfb4e4d9
Implemented facet-based edit operation for real.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@167 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-02 20:33:11 +00:00
David Huynh
512cd16381
Implemented recon by keys, guids, and ids.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@165 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-02 18:19:20 +00:00
David Huynh
99ae6109d8
Started work on key-based recon.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@164 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-02 03:31:58 +00:00
David Huynh
e57aae888b
Hooked up the recon service at data.labs.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@163 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-02 00:33:32 +00:00
David Huynh
f16727c20c
Refactored recon code on the server side to prepare for supporting other modes of recon.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@162 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-01 23:33:23 +00:00
Stefano Mazzocchi
621655372f
- save encoding and confidence in the project metadata
...
- use the saved encoding for decoding
- don't error when fingerprinting null
git-svn-id: http://google-refine.googlecode.com/svn/trunk@160 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-01 04:56:16 +00:00
David Huynh
bc9bc54d30
Implemented a meta parser that looks for a language prefix and picks the right parser.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@159 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-01 01:50:56 +00:00
David Huynh
acfa19a683
Moved GEL stuff (gridworks expression language) into gel package.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@158 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-01 01:30:31 +00:00
David Huynh
7c38fbb945
Created an ast package for gridworks expression language abstract syntax tree nodes. Moved parsing exception class out to its own file.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@156 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-01 01:14:21 +00:00
David Huynh
983be19e14
Made EvalError serializable because errors can be cell values and need to be saved.
...
Turned is* functions into controls, since they have to be able to test errors, and only controls can do that, not functions.
Polished display of errors in cells and in expression preview dialog.
git-svn-id: http://google-refine.googlecode.com/svn/trunk@155 7d457c2a-affb-35e4-300a-418c747d4874
2010-03-01 00:21:13 +00:00
Stefano Mazzocchi
0c6590fe2c
- added an encoding guesser
...
- fixed a bunch of encoding issues
- added a function to reinterpret call content in another encoding
- added a 'phonetic' function to the expression language that supports metaphone and soundex
- updated the COS library to the latest released version
- added the IBM ICU4j library (that contains the encoding guesser)
- added examples with same content but different encodings
git-svn-id: http://google-refine.googlecode.com/svn/trunk@154 7d457c2a-affb-35e4-300a-418c747d4874
2010-02-28 21:51:33 +00:00
Stefano Mazzocchi
d9e67ac806
- diff now can act before two dates (still to be fully tested)
...
- added string fingerprinting function (useful for clustering)
- fixed unicode() function which wasn't returning correct values
- added a toString method to EvalError to know what error that was
- fixed a NPE in TextTransformationOperation
git-svn-id: http://google-refine.googlecode.com/svn/trunk@153 7d457c2a-affb-35e4-300a-418c747d4874
2010-02-28 00:55:09 +00:00
Stefano Mazzocchi
f1923758e7
- add a bunch of new functions
...
- very lax date parser
- lots of new advanced string functions
- new version of commons-lang
git-svn-id: http://google-refine.googlecode.com/svn/trunk@152 7d457c2a-affb-35e4-300a-418c747d4874
2010-02-27 08:56:04 +00:00
David Huynh
25fd5794cd
Added choices blank and error to list facets.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@151 7d457c2a-affb-35e4-300a-418c747d4874
2010-02-27 06:59:55 +00:00
David Huynh
49e7241d1d
Re-organized functions into a few sub-packages.
...
git-svn-id: http://google-refine.googlecode.com/svn/trunk@150 7d457c2a-affb-35e4-300a-418c747d4874
2010-02-27 05:59:09 +00:00