[ANNOUNCE] Auto-Complete Editor
 

I think I've now managed to decode enough of the auto-complete dictionary format to implement a basic editor - it works for me anyway :D

So, I'd like to announce the snappily-named "AutoComplete Editor", coming shortly to an extras-devel repository near you. It allows deletion and addition of terms from the custom auto-complete dictionary. I'm not sure how well this'll handle international character sets, so I'd appreciate some feedback from anyone using those.

For future reference, the custom auto-complete dictionary (/home/user/.osso/dictionaries/.personal.dictionary) format is:

An 8 byte header, consisting of a 3-byte hex sequence 80 00 01 (though I've also seen 01 00 01), followed by a single byte indicating the number of different dictionaries, followed by a single byte indicating the file size (in 256-byte multiples, so 0x04 indicates a 1kb file), followed by 0x00, followed by a 2-byte sequence indicating the position the padding starts.

This is followed by one or more 8-byte dictionary entries, consisting of a 2-byte sequence indicating the dictionary language (or 0x00 followed by a single byte), then a 2-byte sequence indicating the start position of the dictionary entries, followed by a 2-byte sequence indicating the number of entries in the dictionary, followed by a 2-byte hex sequence 00 00.

The dictionary data is stored as a single byte indicating the string length, followed by the string itself. The multiple entries follow straight on from each other, with no other delimiters.

Here's a screenshot for 0.0.4 (there's been no major changes to the default layout since then):
http://www.robinhill.me.uk/share/Aut...itor_0.0.4.png

Current releases are:

0.0.12 in extras-devel. This allows:

• Editing auto-complete dictionary
• Deleting entries with/without warnings, or instantly by selection
• Moving entries to a blacklist
• Editing of entry blacklist
• Auto-application of blacklist without launching GUI (suitable for scripting or scheduling)
• Overriding of character set used to encode/decode dictionary entries

For those having issues, please see this post for how to generate a clean dictionary file. If this doesn't show the same issue then please consider sending me the original dictionary - all received files will be kept completely confidential and will be deleted as soon as I've fixed the relevant issue. Any dictionaries can be emailed to me at: maemo at robinhill.me.uk

Re: [ANNOUNCE] Auto-Complete Editor
 

Sounds good, I will try it as soon as it is available with umlauts :D

Re: [ANNOUNCE] Auto-Complete Editor
 

Sounds interesting and useful, thanks.

If you haven't done so already, I think an interesting option to implement would be for your app to check the "apparently fixed" things, and if they don't match your expected values, asking the user to contact you, just in case these bytes aren't fixed after all.

Re: [ANNOUNCE] Auto-Complete Editor
 

Thanks, thanks, thanks, thanks!

For me the auto-complete feature have been turned completely useless due to misspelled words added to the dictionary, mixing of Spanish and English words, etc.

Re: [ANNOUNCE] Auto-Complete Editor
 

A very good suggestion - I'll look at adding that in.

Re: [ANNOUNCE] Auto-Complete Editor
 

Version 0.0.2 is now available, with this check.

Re: [ANNOUNCE] Auto-Complete Editor
 

Just tried it but got error:

File "/opt/AutoCompleteEditor/AutoCompleteEditor.py", line 17, in <module>
w.loadData()
File "/opt/AutoCompleteEditor/ACE_gui.py", line 157, in loadData
self._dict = ACEFile()
File "/opt/AutoCompleteEditor/ACE_file.py", line 34, in __init__
self.read()
File "/opt/AutoCompleteEditor/ACE_file.py", line 119, in read

Wasn't able to catch the whole stuff.
Any clue?

Re: [ANNOUNCE] Auto-Complete Editor
 

Out of interest, how did you determine the format anyway? It's a shame it isn't documented openly by Nokia.

Re: [ANNOUNCE] Auto-Complete Editor
 

That's an entry count mismatch error - it's failed to read the correct number of entries before hitting the padding start point. Would it be possible for you to email/PM me your ~/.osso/dictionaries/.personal.dictionary file?

Re: [ANNOUNCE] Auto-Complete Editor
 

Basically I just removed the dictionary & rebooted (making sure I was starting with an empty file). I then just use Notes and added a single new word at a time, copying the dictionary file out each time. I then looked through the different files and noted the differences.

From there it's just a matter of trying to figure out what each means - the format used for the dictionary entries was fairly straightforward (once I realised that the delimiter between the words was the length indicator), so it was just the header changes that I needed to look at. Some of those are pretty trivial (the number of entries increments by one each time, so it's clear what that is) but some of it is pretty much guesswork - the file size (in 256-byte multiples) for example - it's a pretty random thing to store and I don't really see why it's necessary, but that's the only thing I could figure it as being.

Re: [ANNOUNCE] Auto-Complete Editor
 

Sorry for being the dummy here, but eactly how can I edit AutoCompletion with this? Or is the point simply that this is a tool for me to edit AutoComplete behaviour now?

Re: [ANNOUNCE] Auto-Complete Editor
 

Well... I actually tried editing it by hand sometime ago ;)
So I guess I'm gonna deleting it.

Thx
William

EDIT: no way. Just tried deleting my messed up personal dict, nothing changed. Same result removing .used_dictionary. Anything else?

Re: [ANNOUNCE] Auto-Complete Editor
 

When you run it, it should load up your current custom auto-complete dictionary (that's the list of terms that you've used in any auto-complete enabled application which aren't in the base dictionary).

You can then select terms from the list and delete them (e.g. passwords that have been automatically added at some point, or misspelled words), or add new terms to the dictionary (this shouldn't really be necessary though, as any terms you use in applications will get automatically added).

Finally, click the "Save changes" button to write out the new dictionary and force the OS to reload it.

Re: [ANNOUNCE] Auto-Complete Editor
 

Probably safer deleting it, yes - you'll need to reboot (or kill hildon-input-method) to force the OS to drop it's cache of the old file though (otherwise it'll just write it out again).

Re: [ANNOUNCE] Auto-Complete Editor
 

same here. what shd I do now?

Re: [ANNOUNCE] Auto-Complete Editor
 

You get the same error with a brand new .personal.dictionary file? In that case, could you send me a copy of it?

Re: [ANNOUNCE] Auto-Complete Editor
 

If possible, could you send me a copy of the dictionary file (/home/user/.osso/dictionaries/.personal.dictionary) via email/PM? I'd suggest checking the file with more, notepad, or something similar to make sure it doesn't contain anything personal first though!

Re: [ANNOUNCE] Auto-Complete Editor
 

No way. I've removed bot personal and used dicts, but still getting errors. :|

Re: [ANNOUNCE] Auto-Complete Editor
 

For anyone getting errors, please send me a copy of the dictionary file. I'd suggest the following process, to make sure that no personal data is contained in the file:

• Backup the dictionary: "mv /home/user/.osso/dictionaries/.personal.dictionary /home/user/.osso/dictionaries/.personal.dictionary.old"
• Kill hildon-input-method to clear the cache: "killall hildon-input-method"
• Launch the Notes application and type in a new word (any random set of characters should work fine)
• Retry AutoComplete Editor to make sure you still get an error
• Copy the new dictionary, which can now be safely sent to me: "mv /home/user/.osso/dictionaries/.personal.dictionary /home/user/MyDocs/failing_personal_dictionary
• Replace your original dictionary: "mv /home/user/.osso/dictionaries/.personal.dictionary.old /home/user/.osso/dictionaries/.personal.dictionary"
• Kill hildon-input-method again to clear the cached data: "killall hildon-input-method"

You can now send me (via email, PM, or attaching to a message here) the /home/user/MyDocs/failing_personal_dictionary file.

Re: [ANNOUNCE] Auto-Complete Editor
 

im going to give this a try...

Re: [ANNOUNCE] Auto-Complete Editor
 

Ok, at last I was able to make the app start. Not it just complains about the file not being in any known format, but it the correctly shows content.

File attached. Just le me know if you need anything else.

Re: [ANNOUNCE] Auto-Complete Editor
 

Same issue as maddler, though I only got to it because I went ahead despite the warning message about the file not being in any known format. Which then resulted in the python error maddler had. Deleting it resulted in the same issue. Let me know if you want mine too.

Re: [ANNOUNCE] Auto-Complete Editor
 

it seemed to work great on first use. but then i noticed it didnt really delete all the stuff i deleted. so i tried to go back to the app and then got all the same errors as everyone else. maybe upon deleting its not properly recalculating the number of entries remain? i think i saw somewhere that you said the number is stored. i used ctrl to delete many at a time. maybe it only reduced the number by 1 when i hit delete?

edit:
also when i was done deleting stuff, i hit save and nothing seemed to happen. i figured it just happened quickly since its just text. but maybe it didnt really save? because all my wrong words still existed even after rebooting first then typing text. i hit save many times since nothing seemed to happen. could that have messed up the word count? it would be nice to get a maemo note 'Auto Complete Dictionary Saved' or something like that.

oh another thing... when scrolling down the list, it would keep highlighing words above the point where i let go of the sreen to stop scrolling. it was annoying. i had to basically do one page of words at a time because of this.

thanks for the app it will be so useful once you get things worked out. i didnt even realize all the misspellings i had!!!!!

Re: [ANNOUNCE] Auto-Complete Editor
 

Many thanks for that. Looks like you've got a 3F instead of 03 in the header - I wonder what that signifies :confused:

The rest of it looks the same as mine though, so I guess I'll just make note of the value and write it back out when saving (in case it's important somehow).

Re: [ANNOUNCE] Auto-Complete Editor
 

If you could do, yes please - the more examples I have, the better the chance of actually figuring out what the format is.

Re: [ANNOUNCE] Auto-Complete Editor
 

I don't know if you know this, apologies if you do, but it seems like there's a limit imposed by HIM on how many words can be stored:

Re: [ANNOUNCE] Auto-Complete Editor
 

No, it does a count of the entries at the end, so there's no risk of that being the issue. Can you send me a copy of the file that's giving you the errors?

I'll look into adding that - thanks for the suggestion.

That must be a problem with the multi-select functionality - I'll have a look at whether there's anything I can do to tweak that.

Re: [ANNOUNCE] Auto-Complete Editor
 

I knew there must be a limit - there's only 2 bytes for the number of entries, so logically there's a 65k limit on entries. If I'm right about the one byte for the file size, then there's also a 64k limit on the that (hmm - perhaps there's only 2 bytes for the padding offset then - there doesn't seem a lot of point in having 3 bytes if the maximum will fit in 2).

I'll have a play around with generating a very large dictionary and see whether I can hit the upper limits (and whether they help tell me anything about the format).

Re: [ANNOUNCE] Auto-Complete Editor
 

i also got an error

Traceback (most recent call last):
File "/opt/AutoCompleteEditor/AutoCompleteEditor.py", line 17, in <module>
w.loadData()
File "/opt/AutoCompleteEditor/ACE_gui.py", line 157, in loadData
self._dict = ACEFile()
File "/opt/AutoCompleteEditor/ACE_file.py", line 34, in __init__
self.read()
File "/opt/AutoCompleteEditor/ACE_file.py", line 139,

what should i do??

Re: [ANNOUNCE] Auto-Complete Editor
 

Please see this message for how to generate a clean dictionary file. If you can then attach it to a message here, I'll take a look at why it's failing.

Re: [ANNOUNCE] Auto-Complete Editor
 

I've just uploaded version 0.0.3 to the autobuilder, so it should be in extras-devel shortly. The changes are:

• Information popup when the file has been saved.
• Uses multi-selection instead of extended selection (just tap to select each additional value, no using ctrl or shift)

Additionally, there's now an extra byte read from the header (and written out when the file is saved). This should fix the loading error several people have had (and thanks to those who've sent me a copy of their dictionary files). I've still no idea what this byte signifies - it's obviously not static but as nothing else is different, it also doesn't seem too important.

Re: [ANNOUNCE] Auto-Complete Editor
 

I still have the same error with the new version or do I have to reboot after the installation? :(

Re: [ANNOUNCE] Auto-Complete Editor
 

Thanks for sending me your file. Unfortunately it's in a totally different format :mad: Would it be possible for you to follow the instructions here and send me copies of the dictionary file after adding 1, 2, and 3 terms? I really need some extra data to figure out where the key fields are in this format.

Re: [ANNOUNCE] Auto-Complete Editor
 

I could do that but I thought that it would be more useful to send you a "real" file.

Re: [ANNOUNCE] Auto-Complete Editor
 

It is, yes, but I now need several files with different numbers of terms, so I can see where the changes are happening. As I said, your file is in an entirely different format (with a 32-byte header instead of the normal 16-byte header), so it's not just a case of reading in an extra byte or two (which is all that I've needed to do to get the other files working).

Re: [ANNOUNCE] Auto-Complete Editor
 

Hi,
after installation I got this error:

Traceback (most recent call last):
File "/opt/AutoCompleteEditor/AutoCompleteEditor.py", line 17, in <module>
w.loadData()
File "/opt/AutoCompleteEditor/ACE_gui.py", line 165, in loadData
self._dict = ACEFile()
File "/opt/AutoCompleteEditor/ACE_file.py", line 39, in __init__
self.read()
File "/opt/AutoCompleteEditor/ACE_file.py", line 149, in read
raise ACEFileReadException('Entry count mismatch')
ACEFileReadException: 'Error reading dictionary: Entry count mismatch'

After I moved my .personal.dictionary and typed a few words in Notes app. the application started OK but it doesn't display cyrillics in the right encoding, so I couldn't tell one Russian word from another.
I attached the newly created file just in case. Let me know if the original file could be of use.
Attachment 11050

Re: [ANNOUNCE] Auto-Complete Editor
 

Thanks for that. The format looks standard at least, so I just need to look at how to decode the data into cyrillic. I'm not sure why your original one came up with the mismatch error though - perhaps you could post that as well & I'll have a look. It may be I need to relax some of the checks.

Re: [ANNOUNCE] Auto-Complete Editor
 

Please use XB-Maemo-Display-Name field in debian/control to beautify the name (e.g. Auto-Complete Editor). I had a surprisingly hard time finding this app after reading its name in the thread title.

Re: [ANNOUNCE] Auto-Complete Editor
 

Sorry, I've been using py2deb, which seems to be missing any way to set this field. I'll switch to editing the debian files directly in future.

Re: [ANNOUNCE] Auto-Complete Editor
 

Woohoo! I've had a bit of a breakthrough :)

The dictionary file can actually hold multiple dictionaries (if you change the dictionary used in Settings). That explains some of the differing file formats I've been sent.

This is going to mean a fair bit of reorganisation/redesign needed, and I still need to figure out how to decode the dictionary language ID to a language name (and text codec).