Meeting about Prepostition Error Correction and Syntax-matching Analysis

james — Thu, 13 Nov 2008 12:00:00 +0000

Thoughts before meeting:

Do I need to determine semantic categories (Boston vs MIT)?
If so, I need to expand matching to give some sort of score based on the closeness of the match (perhaps using previous WordNet code)
How does syntax-based matching help? Isn't immediate context most relevant?
Google n-grams approach is different to any currently published work

Discussed in meeting:

Don't bother with sem. cats. yet, just assume exact matches.

read more

Potential Progress

james — Fri, 17 Oct 2008 14:46:09 +0100

Just met with Steve. Still need to sort out bug in Moses which is preventing insertions, and will do some analysis on current system with large chunks to see if it's working for those, and if not why not. Also discussed potential third chapter - using what I've got to do some paraphrasing, extending C C-B's work to find non-contiguous paraphrases. Will do reading this week to see if there's a good opportunity here.

Lost Phrases

james — Tue, 09 Sep 2008 20:04:21 +0100

I started this morning hoping to sort out the issue whereby spans were not being split up correctly. First, I realised that this website isn't backed up at all, so I set up a script to sort that out. Then I got back to work. The issue is that matches are currently split into contiguous spans based only on whether the input words they match are contiguous. But what if the input words are contiguous, but the matched words in the example are not? We need to split this into multiple spans. Further, what if the words are contiguous in both the input and the example, but the ordering is changed?

Dependency Matching

james — Mon, 08 Sep 2008 16:01:38 +0100

There was a problem with the dependency matching algorithm which resulted in some bad matches ranking more highly than they should. The algorithm allowed a single dependency in the input to match several dependencies in an example, provided they all had the same lemmas and relationships.

Switch back to Moses alignments

james — Wed, 06 Aug 2008 13:43:57 +0100

Having abandoned the idea of using the GIZA++ alignment scores, I can now switch back to the more reliable Moses alignments (a smart combination of both GIZA++ alignment directions). Having done so, the BLEU score jumped about 3 points:
r157-20080731053726: BLEU = 20.05, 52.9/27.1/16.8/11.0 (BP=0.885, ratio=0.891, hyp_len=4809, ref_len=5398) r169-20080805193448: BLEU = 23.21, 52.3/28.2/18.3/12.4 (BP=0.966, ratio=0.967, hyp_len=5219, ref_len=5398)

Repeated words in output

james — Wed, 06 Aug 2008 13:34:43 +0100

I thought I'd fixed the problem of target words appearing more than once in the output, but I hadn't done enough. While I had prevented a word from appearing twice in one phrase, it could still get into the output twice by being aligned to words in two separate but <linked> spans. Fixing this required a bit of restructuring, as generation of target phrases needed to be more aware of what was going on elsewhere.

Script updates

james — Wed, 06 Aug 2008 13:26:26 +0100

Updated do_everything.sh:
- includes support for running EBMT only or SMT only
- can use an existing results dir as a base when running SMT only
- cleaned up params file

Updated make_alignment_images.pl:
- includes support for Moses alignment files
- includes basic range support (-from and -to)

Debug Threads

james — Thu, 31 Jul 2008 05:50:08 +0100

I modified the EBMT system so that when in debug mode it doesn't hang around waiting for diagrams to be created. Before the change, CPU utilisation was down around 40% for the java process, because it would wait for deps2dot.pl and dot to run before resuming. I've changed it so that diagrams are created in their own thread, which are spawned and then forgotten about. Now the java process is consistently running at 100%, with about six or seven other threads creating images simultaneously, using up roughly another 100% CPU.

Per sentence BLEU scores

james — Thu, 31 Jul 2008 00:03:55 +0100

New script, sentence-bleu-scores.pl, generates an HTML page which displays per sentence BLEU scores for several systems, and their difference from the base system score (i.e. Moses). Can be sorted on any column using JavaScript. SHould help identify problem sentences.

Removed GIZA scores

james — Fri, 18 Jul 2008 20:42:34 +0100

Tried running the system with the GIZA++ alignment 'probabilities' removed. BLEU score barely changed. Changing XML mode from exclusive to inclusive gave a big performance jump though. Looks like I need to play around with the probability parameter some more.