Okay, this is weird. It is definitely the same exact binary, segfaulting on the r&d-virgin N900 at bootup, but not on the development one where it no longer does.

So there is still more work do be done, I am missing something somewhere. But for now, I am out. Sleep. Needs to happen. Besides that, I updated the source once again in case the above post didn't make that obvious. I cleaned up the code that handles the R&D-virgin N900s correctly (there was a cal_finish() call left in one of those code paths from years ago, and I fixed the messages it provided to be more meaningful thanks to me getting new understanding of what they mean myself, and made it actually properly distinguish between the 'R&D virgin' error and the other errors that might be returned - I also made it use the cal.h defined macros for the return values of cal_read_block instead of just going by the numerical value as the old code did..

Edit: P.S. reinob, etc, anyone still left with an N900 that hasn't been R&D mode turned-on, ever?

Sure, me, make myself sleep when I have a problem like this.

I know what it is, damn simple: the /dev/shm directory needs to exist before you can mount on it. Apparently, by default, that early in the boot it doesn't.

The reason I didn't notice is that 2 days ago on my dev. N900 I mkdir'ed that directory from inside the @boot console. Since then it hasn't disappeared. But I guess normally is does go away between reboots? Really odd to me. I am done digging for a few days. The solution is right there, but I don't want to just add mkdir and rmdir calls to the code without knowing if that's the 'right way'. I want to know how/why /dev/shm normally gets created and then disappears - and why manually making it at book kept it safe from getting killed through the reboots. Is it some sort of virtual directory that isn't actually written to the flash, normally? If so, I'd rather figure out how to do it properly that way and not chip away at the NAND flash with a a pair of mkdir/rmdir calls every time rdmod runs.

Okay, NOW sleep.

I believe my 1st N900 never been in R&D-mode, but I am not sure. But hey, it's worth the try right?

@Mentalist Traceur,

/dev is actually not stored in the filesystem, but also a tmpfs.
Going back to /etc/init.d/rcS (I have to someday post details about the whole /etc/init.d in the Maemo5 boot process thread, which only dealt with the upstart part).

it begins by doing start_udev(), which does
In prepare_start_udev the magic happens:
(tmpfs_size = 1MB unless set in /etc/udev/udev.conf, which DOES set it to 10MB)
(note that *my* /dev currently uses up 88.0K, so I'll see if I can modify that back to 1MB (or even less) without breaking anything, and hopefully giving some RAM back to Maemo

IN SHORT:
if you run your program during your early boot console, rcS has not run yet, so udev is not running yet and /dev is merely a directory of your rootfs, where you can store your p0rn if wanted (and your semaphores

Once rcS runs whatever you had under /dev is not visible anymore.

[EDIT] so yes, feel free to mkdir -p /dev/shm before rcS runs, this way nothing should segfault

[EDIT**2] I just modified /etc/udev/udev.conf and changed the size to "512k". Maemo booted fine (this is my "production" N900). Obviously I cannot tell if less actual RAM is being used or not, but I internally feel better knowing that less RAM-management is taking place

Hi again,

I have just had a look at @freemangordon's implementation of libcal, and it's a bit weird.

First of all: cal_read_block() and cal_write_block() do *internally* cal_init_ and cal_finish_ (which do the actual set-up and free()ing of the buffers).

So from the point of view of a user application you do not need to call cal_init() or cal_finish(), as they are nops.

The first thing your program does is to read the r&d_mode block. After the call to call_read_block the only thing that is valid is tmp_ptr and len, i.e. cal_s has been freed.

So towards the end of the program, when you actually call cal_write_block (if the mode string is different), cal_s is undefined, but it's OK because it will be cal_init_()'ed again.

If I read correctly, when you want to clear r&d_mode (*rd_mode_string == '\0')) you use sizeof(rd_mode_string) as the length. However the length should be zero. sizeof() gives the actual size of rd_mode_string, which is the same as sizeof(char *) which is namely 4 (for a 32-bit computer). You actually want a zero in there.

Basically you want to drop the "if(*rd_mode_string == '\0') and just use the normal path, where you use strlen(rd_mode_string) (which will be zero when rd_mode_string is empty anyway), however you need to get rid of the "+1" (why is it there anyway?, that's going to cause random segmentation faults).

But cal_write_block_() has a problem when len = 0, because it does malloc(len), which when len is zero may return NULL or a pointer. If NULL however, the function exits with an error (CAL_ERROR). This is something for @freemangordon to fix, unless it is required that when writing len > 0, meaning there should be another function to erase a CAL block.

I've printed cal.c and will read it when I'm on the train. Maybe I spot something.

It is not "my implementation" but a RE, so whathever weirdness you found, most probably it is Nokia legacy . I even fixed some obvious bugs, but not all of them.

I disagree, those are part of the API and the implementation may change in the future versions.

Correct. And this comes from the Nokia (I'll double-check with IDA again). I think libcal was never meant to be used for block erasure, that is why len == 0 is not checked for. On the other hand it is possible that malloc in our glibc to return non-NULL for zero-sized allocation, so there is no (real)problem. However, len == 0 case should be processed correctly(it is going in my todo list, along with packaging).

Ah OK. I didn't know it was RE. I thought it was more "clean-room". But yeah, looking at the mess cal.c is, it lacks the elegance one would expect from carefully written code [ shame on you, Nokia :]

Repeat after me: WE are the API

Thanks. By the way, I just verified (on a N900 having R&D previously set) that setting and clearing a flag using Mentalist Traceur's program (I used serial-console as example) works fine. Double-checked with cal-tool and everything looks OK.

I've compiled libcal with -DDEBUG and run "rd -q". I've attached the output, in case somebody is interested.

I find interesting to see which blocks are stored in the cal (note: libcal is effectively the implementation of a file system).

Will try to spend some time figuring out the content of those blocks.

- TL;DR -
Source in front page updated; deemed stable and fully working. Further work - NOT in the near future, will be refactoring of the code, and some logic improvements, namely splitting the parsing of the command line parameters and the actual building of the rd_mode_string. Thanks to everyone who commented as your comments are all either helpful or appreciated. I disagreed with reinob on one point (the one about the '+1' after the strlen when calling cal_write_block()).
- END TL;DR -

None of the above future work hopes really matters as far as normal functionality is concerned. I.e. the logic improvements only matter for performance if you're an efficiency zealot (so like me): the rd_mode_string is rebuilt way more than it needs to be.
'rdmod -e -s no-omap-wd no-ext-wd' will invoke 3 realloc() calls. So it is better to abstractly to through the arguments and build a compact set of what the arguments want, which is just booleans/ints/chars/bitfields/whatever, not wild dynamic memory thrashing as with the current way, then calculate the final string size, malloc once, compose the string - just once - then write. But it's not like the way it is currently is even remotely noticeable as slow though...

Well, if you ran my current first-post source code (well, the compiled binary thereof obviously), it would tell you when you ran stuff (something like "R&D area blank... maybe you never set R&D mode." but worded better), if you never did have it in R&D mode. At this point I tested it on my R&D-virgin N900 enough to know it works though, so further testers aren't as desired as I was hoping for when I made that post (I think I was just longing for sleep and kinda implying "someone go test this so I don't have to"...but then I stayed up and tested it anyway last night, and by the time I've written this post I've gone through even more final testing).

Yes plz. Also, this was helpful. I'm snipping the rest, but I REALLY need to get into the habit of checking those scripts. I did earlier today when I came back to digging into it about [strike]two[strike] nine hours ago: It proved helpful in other ways.

Yeah, this was definitely helpful in starting me down the right path. As you say, once the new tmpfs /dev is mounted over the root ubifs or whatever /dev, all the relevant devices nodes disappear.

I wanted to go a 'cleaner' route than simply mkdir-ing a /dev/shm, let alone mkdir-ing it and removing it every time (like a ninja - not leaving any traces in your rootfs is I don't have to - you'll never know I was there - or that rdmod was there in this case).

This presented the next problem: I had to mount the temporary /dev like the N900 did at boot, then mkdir on that. But now the device that cal resides on, /dev/mtd1, disappeared and needed to be somehow brought into the new /dev - fortunately, mknod is a thing. So that's been my process today - getting the code to check if /dev is mounted as a tmpfs filesystem, and if it's not, mount one, then mkdir /dev/shm and mknod /dev/mtd1, and go from there.

I've got that all working fine now, although I've seemingly created a much more scary bug just now - I seem to have somehow straight up wiped the cal area (lock code reset to 12345, r&d settings being seen as R&D-virgin after every write... Woot broken code.) Not sure what the hell I managed to do there, but that's not exactly a good thing. I'll be debugging that later as time permits lol. As typical with bugs in this program, they entail things I wouldn't even know where to begin the investigation of, lol. Correction, before I even made the post (this post has been open for hours... every hour or so I come back to it). I seemingly have fixed it, without fully figuring out the cause. Possible contributing factors:
I was 'fusing' the rd_mode_string and rd_mode_current pointers (setting them to equal each other) in a spot. I don't think this could've initiated the problem because it only effected the R&D-virgin code path.
I was failing to properly unmount/unwind the stacking of tmpfs that I was creating - I accidentally left a copasted "umount" statement in without actually editing the parameter so it was trying to umount /dev/shm twice. This was unlikely to be the issue because that problem doesn't surface until you try to do other normal tasks (like continue booting) - then the kernel flips a **** and panics because /dev is suddenly empty. Really bad that I slipped up like that (but that's why I test before release, to catch dumb **** like that), but the fix was trival and I don't think that was what was causing the issue with the cal area self-wipe either. *Shrug*

Anyway, I fixed those things, double checked that everything else was seemingly okay, recompiled and the issue seemingly went away, so I don't know what exactly it was, but it was weird and potentially serious if it ever came up again. In fact, I just enabled R&D mode for the first time on my main use-as-phone N900, from inside my boot shell, no problems, WOOOOOOOOOO **** yes.

Good to know. I'm not sure that the RAM management is any smaller - I thought tmpfs dynamically resized itself anyway, with the kernel only pushing things out when it needed the room? (As opposed to ramfs which reserves the whole chunk... maybe? I have this VERY hazy memory to that effect from somewhere...)

This is all very informative, thanks for taking the time to dig deeper. For the time being I'll admit I've concerned myself only with the parts of libcal that weren't working on one way or another, so it's nice to see a general critical look at some of it.

You're absolutely right about the sizeof() thing. I've got no idea what I was thinking when I did that - it obviously harkens back to 2010-2011 when I didn't really know what sizeof() truly meant. Good catch. I just tested it with 0 and it seems to work just fine.

Actually, here you're off - there is a very specific reason for the +1: what's a C-string depend on? Terminating null. What's strlen() return? Length without the terminating null. What happens if we write only strlen chars? We omit the terminating null pointer. Now, I went ahead and looked through the code path part of the way into cal_write_block(), into set_header(), and it looks like the CAL block structure DOES store the length (so it's not relying on a null byte to tell it when the block ends - great).

But consider this: what happens if I don't write in the terminating null. When my code goes to load the string, it will get a len that's just long enough to fit the string content, without the c-string null terminator. It will also get a void * containing said string, which it then proceeds to cast to char * and treat as a string.

The outbound end would have to be tweaked to get the string out of cal without a terminating null, which means more work to reassign it to a new char array one size bigger, to fit the null byte. That or the string.h functions become useless without further work.

If I recall correctly the process that went in to the early coding on rdmod 2-3 years ago, then that + 1 is the precisely the result of originally using strlen() by itself, and consequently reading random garbage past the end of the written-in string (and consequently writing trailing garbage back in some of the time). In fact, I'll even put it this way (in a friendly manner mind you, I'm not angry or lecturing here): I challenge you to remove the +1 and see if any segfaults (of which I presently see none, and I assure you I test reasonably thoroughly) disappear, and more importantly, if you see any segfaults or other issues appear.

Now that said what WOULD be a good reason to take the approach of not writing in the c-string terminating null, is if you set R&D mode settings on using Nokia's Flasher, then try to read it with R&D mode control and that gives problems (this being completely not caused by the +1 but by the fact that the strings coming out would have no terminating null for my code then (or the standard c-string functions thereof) to stop at), then I will, for the sake of compatibility and consistency, write the strings without the terminating null to the cal block and compensate for the lack thereof in my own code when pulling the string out.

This to me seems to present a stronger reason in favor of writing strlen()+1, even in the empty case. Which is what the latest code does. Eliminates the separate code path (like you suggested) but keeps the + 1.

Hope that went well for you. I'm always happy to learn so feel free to post and other findings you find interesting.

I think libcal does have a call or two for block erasure, I could've sworn I saw functions titled something like that in there.

I actually would love it if some more formalized effort occurred to standardize some changes/differences/features where appropriate - but I'd want it to actually have some sort of formal way of reaching consensus - random developers simply deciding "this package will do /this/" is fine in general but helps make things less cohesive when it's fairly system-integrated libs we're talking about.

Oh yes absolutely. rdmod can and will erase your R&D setting into being a blank string, and otherwise clearing just individual flags, just fine. That functionality was around and fairly vetted for a pretty long time now. But that said, it's always nice to have more confirmation/testing.

That's a very generous title for what is a basic linked list with some metadata and actual data per node/link. But yeah, in a sense it is. It's worth noting that it is clearly sequential in nature, whereas a filesystem can typically locate files at random pretty easily and efficiently. Filesystems usually don't need to potentially traverse their entire contents to find what they are looking for - libcal may well have to.

Now that libcal is open I may well just write a program to simply allow perusing the data and viewing the data of each block. Unless someone's already done it.

...and if I have time. Given how long it took rdmod to reach the full functionality I wanted out of it to begin with, there's no guarantee I will have that done any time soon - and in fact, I will explicitly say I will do my best not to for a while because I just used all of my free time these last three days or so doing just rdmod work and I have to make sure everything else is taken care of.

Would it be too much to assume that CAL in N9 is of similar structure as CAL on N900...?
Even as it's locked in Open Mode on N9 it would still be possible to read it, just for intrest of the content.