This. Right here. You are awesome. That was the missing piece.

For everyone else who's curious, my understanding is: Semaphores need a spot in shared memory that they can be stored in (it can be shared either between different threads in a single process, or between different processes - libcal uses the latter). I guess the default location for that is whatever virtual filesystem /dev/shm is mounted as. At the stage were my preinit shell prompt runs, /dev/shm is not mounted, so the sempahore creation fails.

Thankfully, /dev/shm is a simple tmpfs filesystem in typical Linuxes, so "mount -t tmpfs tmpfs /dev/shm" as root is sufficient; then rdmod runs with no segfaults!

Furthermore, now that I understand this, I can understand more of the straces of rdmod vs. cal-tool. It seems that cal-tool, upon detecting that /dev/shm is missing (it doesn't seem to be catching the segfault, rather I assume it actually checks for it somehow), it seems to call mmap, after which it creates the semaphore in /tmp/ instead of /dev/shm/. (This is an educated guess interpretation based on the fact that there is a stat() call on /dev/shm, followed by an mmap() call, followed by a stat() on /tmp, whereas rdmod just does the first stat and then segfaults. My strace reading skills are still in their early stages.) [Edit]Correction, it's statfs(), not 'stat()', that it calls on the directories, and more importantly, the mmap does not seem to be the thing that matters in this situation, as a closer look shows there's a bunch of mmap calls all over the place leading up to the segfault and I don't think that that one mmap has anything to do with cal-tool managing to switch to using /tmp as a spot for its semaphore.[/Edit]

(In retrospect last year when I was trying to figure this stuff out with strace on cal-tool vs my rdmod I should've known to look into /dev/shm (as it gets stat'ed right before the segfault, which I had seen last year) but at the time I had no context to know what it meant, and I think eithout understanding that semaphores were used I'd've prob'ly not realized the significance of anything I found at the time even if I did think to attribute any significance to /dev/shm.)

The next few steps are to figure out how to either mount /dev/shm from rdmod when needed, or I think even better, do what cal-tool does and use mmap somehow.

What really makes this confusing is that I can't find any documentation explaining WHY /dev/shm is the default location where named semaphores go, or how to move/create your named semaphores elsewhere (like cal-tool does). So I don't even really know where to start, when it comes to figuring out how the cal-tool code works to achieve what it does.

Besides that, here's a headscratcher (although the question is just a pure why rather than it getting in the way of a solution at the moment): /dev/shm DOES show up as mounted if you run 'mount'. rdmod segfaults. Then you run 'mount -t tmpfs tmpfs /dev/shm'. /dev/shm shows up in the same exact way in 'mount' output again. No change what so ever. Yet rdmod stops segfaulting.

I want to know how @ paragraph 1 and why @ paragraph 2. But for the time being, I am content with the fact that some way of making rdmod finally work at early boot is possible.

Partial answer to my own question: /proc/mounts is where the actually accurate mounts data is, 'mount' with no arguments seems to print out a list that doesn't reflect the current mounts, at least at that stage of boot. *Shrug*

You just opened the "/etc/mtab vs /proc/mounts (or /proc/self/mounts)" pandora box

The actual information regarding mounted filesystems (as seen by the kernel) is in /proc/mounts (or /proc/self/mounts if you want to be chroot-compatible).

The program "mount" writes to /etc/mtab. The program "umount" writes (deletes lines from) /etc/mtab.

Normally /etc/mtab contains the same information as (or rather, information consistent with) /proc/mounts.

However if the computer crashes it doesn't get a chance to delete /etc/mtab, so upon reboot /etc/mtab will contain stuff that possibly is not relevant anymore. For that reason some people make /etc/mtab be a symlink to /proc/[self/]mounts. However this breaks other stuff.

Ideally /etc/mtab should be wiped during (early) boot, i.e. before anything is mounted. I don't know (at the moment) if/how Maemo5 handles this.

EDIT: In /etc/init.d/rcS you have
and a bit later
and for completeness, if I look in my /dev/shm I have a 20-byte file called "sem.nokiacal" containing, in hex:
"01000000 00000000 00000000 00000000"

/usr/bin/cal-tool uses libphread.so, so I guess it uses sem_open(3) to create a named semaphore. It is (I think) the kernel which takes care of creating the named file under /dev/shm.

so basically sem_open("/my_semaphoe", ...) will create /dev/shm/sem.my_semaphore

Oh okay, that explains that. Thanks.

Ohhh, okay. That's why my /etc/mtab and mount output is completely meaningless - as people might recall my 'at boot' shell of choice is literally virtually the first moment it's possible to boot during. /etc/init.d/* doesn't happen until later.

I'm particularly glad you looked up the exact mount command that's run to get /dev/shm up - my current solution to the segfaults seems to work regardless of it, but if it ever seemed to be an issue, that could come in handy.

On this point you're absolutely right, but this wasn't in question anymore: In case you missed it, freemangordon got us an opensource recode of libcal, which means we can look at what the (presumably nearly identical) version does here:
https://gitorious.org/community-ssu/libcal/

So from that code we already knew that it was using a semaphore, calling sem_open(), etc. That's how I even got to the point where I knew it was segfaulting at the semaphore creation level, which is what led freemangordon to give the hint that he gave that solved our segfault troubles.

But that's not the question. The question is the behavior of cal-tool at that early boot situation where /dev/shm is not yet mounted. Somehow, it still manages to create a semaphore, located at: /tmp/sem.nokiacal

Nonetheless, your statement says more concisely/clearly what cal-tool and other libcal programs do with regard to semaphore use, and either way this remark is appreciated just as your prior ones were.

---Regarding updated source---
I am hoping to have the updated source out in the next 24 hours - it was technically ready to go as it was but it wasn't the best way to do it (I decided to not screw with detecting /dev/shm and simply mount a tmpfs on /dev/shm before cal_init() and umount it after, and it worked just fine, but in principle it's better to actually check if there's a /dev/shm already mounted, and only mount a new one if needed. Until this post I figured /etc/mtab, mount, and /proc/mounts was a crapshoot, but now that I know that /proc/mounts is actually word-of-god (s/god/kernel/ same thing for our purposes), I figured it's worth it to go ahead and see if /proc/mounts has a /dev/shm in it already, and only mount/unmount a new one for the duration of the program if not.

So once THAT is coded up, I will post an updated source, finally.

Code updated in the main page.

It works. It finally works. [edit]In every way that I have always wanted it to that is - it obviously worked almost entire before, but now the segfault at early boot issue is solved too, so it fully works as I've wanted ever since I started on this like 2-3 years ago.[/edit] I have one more N900 with a never-touched-R&D-mode-area, on which it needs a tiny bit of testing (to make sure nothing I changed between last year when I fixed that problem and this year when I fixed this one, or nothing that I forgot to clean up, causes any issues there). But I am pretty confident that even as is, it's usable in all of the scenarios it needed to be usable in.

It's frakking awesome. It scans through /proc/self/mounts to check if a /dev/shm is listed in there, and if it is, proceeds. If it reaches the end of file without having found one, it mounts one before proceeding and then unmounts it later when it's done (only if it mounted it prior of course). For academic purposes I still want to know how cal-tool redirects its semaphore placement to /tmp, but I like my way better unless their way is vastly more efficient.

There are is at least one spot where I want to go over the code to A. verify that I didn't mess anything up with the R&D cal area 'virgin' N900s, and B. tweak the actual output that it makes. But I do believe that should work. (expect an edit within 24 hours (probably, who knows with me) to the front post once more, but after that it should stabilize for a while again)

In the long run, there will be refactoring. What people might not realize is this was effectively my first meaningful C program. It has grown along with me over the years (and, quiet poetically, it is the first piece of code where I used a goto (two even); I've always been against the anti-goto dogma, being an efficiency zealot deep down, but I have also always held off on using it until it seemed like it was notably the best option for what I wanted. In a silly, irrational way, I am glad that it was this piece of code that it first came up).

Anyway, I would ultimately like to make the commandline parameter parsing more robust and more informative when something wrong is done. I would also like to make it do a full pass over all of the commandline parameters before beginning to construct the actual string to write into the R&D mode area of CAL, so that it doesn't was time realloc-ing and strcpy-ing all over the place only to reach later commandline flags that negate what the earlier flags specified. (Not that that's a good use of the program anyway so that's the user's fault, but there is a few possible spots where being able to parse the parameters in advance may in theory be useful.

I also need to get around to submitting the bugfix(es?*) to freemangordon's libcal.

*I noticed this before but at the time didn't understand why it might be an issue: Nokia's libcal uses a named semaphore "nokiacal". The open libcal uses a named semaphore "nokiacal3". So far it seems that nothing that uses the cal area on the N900 does so by other means than libcal, but if by some chance some system/program has "nokiacal" as the named semaphore hardcoded into it, and doesn't go through libcal, we could have a problem of concurrent access to the cal area, which is exactly what the semaphores are presumably meant to protect against. Furthermore, just strictly speaking a recode of some system library (or in general any software) doesn't need to make changes to the name of a semaphore that it uses (or in general, anything it uses) unless there are technical reasons why that's an improvement.

P.S. I expect tears will be shed by those reading my code who are non-believers in the goto. The goto will forgive you, and welcome you with open arms when you come to accept the goto wisdom. (What I seriously mean to say is, those who have a problem with where I used goto calls are ultimately welcome to give me a rewritten version that is qualitatively easier to understand, and we can discuss from there. I am not likely to be convinced by just the typical anti-goto mentality though if it's just preachy.)

I renamed the semaphore while testing the library and forget to rename it back before pushing to gitorious. feel free to make a merge request on gitorious fixing that.

also, libcal is missing debian packaging, I guess we'll address that when the time comes (neo900 )

Thinking out loud here about the issues with the 'virgin' N900s originally having had issues with cal_read_block:

cal_read_block invokes find_block_type, which then returns a struct cal_block *, which if null, causes cal_read_block to return -2, which is what it returns on 'virgin' N900s.

find_block_type then does a bitwise & against a CAL_FLAG_USER (as it so happens, we do pass in that exact constant in the last field of cal_read_block) which determines where it looks in the cal area for the relevant block it seems. It then calls find_block.

find_block simply iterates through the struct cal_block *, which is a linked list basically. It heads onwards through until it finds a block with a name that matches the name passed in. The difference between CAL_FLAG_USER and not is which of the block lists find_block_type selects from the cal area - there is a user_block_list and a wp_block_list variable inside the main cal struct, and those are presumably heads to the lists. I suspect this is largely irrelevant to my situation, but may be worth understanding more what the two lists mean later (i.e. what belongs in which).

I think I will write code some day to peruse these cal area lists and see what's where. I make the tentative conjecture that the cal area is 'sprayed' over some region of the NAND flash and that new settings are added on to the tail of the linked list instead of overriding old ones (I say this because there is a comment implying that if multiple blocks with the same name are found, the one with the highest version must take priority, suggesting there may be more than one "r&d_mode"-named block in the cal area at a given time). But maybe I am wrong, hence the interest in eventually digging deeper.

Anyway, I think the case is pretty clear then: a 'virgin' N900 which has never had R&D mode settings toggled on/off simply has no block in the user_block_list named "r&d_mode". The linked list perusing code of find_block goes all the way to the end until the pointer becomes null, at which point it returns it (that's all it is, a while(block) {} loop, with block being the struct cal_block *, and inside the loop it breaks if the name matches the one requested, or iterates by doing block = block->next. When the loop is over, it returns whatever the current block value is, in this case null).

That is returned by find_block_type without any further consideration. cal_read_block checks if it got a valid pointer, sees that it didn't, and returns a predefined negative value to signify the error.

And having walked through that, I think I have a pretty good idea of how to proceed with finalizing the code cleanly (I don't want to ignore the other error it can give or hypothetical future implemented errors, just that not-found-block one is safe to disregard).

Quick note, segfaulting at boot on my main, still 'r&d-virgin' N900, not sure what's up. I quite possibly just f'ed something up copying it over from my development N900, but this might indicate a deeper issue and that my code is not as ready as I had hoped.

[THIS IS A PASTE OF THE END OF MY FIRST POST, FORUM DEMANDED THAT I TRUNKATE IT DUE TO THE SOURCE GETTING TOO LONG]

Compiling
I coded and compiled and tested all of this on the N900 itself - there, I compiled this using The "-lcal" flag is needed to get the libcal functions to compile in correctly. I did not try using "-Wall", as far as I remember, so if anyone sees anything they don't like while using the "-Wall" flag, speak up, by all means.

But I'm don't wanna compile, give me a binary, or better yet a .deb file
I didn't want to compile back when I first had to either, but I figured it out and my life is better for it (and I never intended to program back then, I just really wanted aircrack-ng and qwerty's gui equivalent of this). I'm not offering it in a precompiled binary form right now because this is mainly for power users, and because I don't really feel like packaging it and sticking it into extras-devel right now (not sure if I could, given the dependency on Nokia's libcal and build-dependency on libcal-dev), however, I will happily place it inside extras-devel (with an all-caps warning in the description, possibly in a non-user category), when I have more time, so long as no one finds horrible flaws with it. Experience using qwerty's program for around almost a year suggests to me that this is a perfectly safe way of modifying R&D Mode flags, but in spite somewhat heavy testing it's possible I missed something. I would also like to see if the -w command is safe. As far as I've seen, correct flags separated by commas are read, extraneous text after that is ignored, but I haven't messed with it too much, or tried writing in strings longer than the R&D CAL area was presumably intended to take at the most (see below, where I talk about -w).

I should probably also tweak the output of -q to be a little more at-a-glance readable.

Why would you want to use this over or along with qwerty12's, if you can compile both already?
The only problem with qwerty12's was that it was a GUI-only program. This is good enough for most uses, and is a bit more intuitive to use, but in my experience, the most crucial (not necessarily most common, but definitely most crucial) situation where you would want to enable R&D Mode on-device is if you're having some problem early in the boot process, and suspect that changing the R&D Mode flags (to disable watchdogs or lifeguard-reset, or boot in spite of getbootstate complaining about the battery even when you know the battery to be a reliable one) will let you boot up, either to repair the initial problem, or leave the device in that state.

Out of the box, the N900 can't get a shell early enough in the boot to do that anyway, BUT, with the addition of framebuffer console to the latest power-kernel versions (thanks to pali), and the use of it to get at-boot recovery shells by pali and myself, the possibility of being able to edit R&D mode flags from on-device in the earliest stages of boot-up is very feasible.

However, since qwerty12's R&D Mode Control was GUI-only, it simply cannot do anything when ran from a console in a framebuffer environment. Also, the use of a command-line tool makes for convenient opportunities to add a couple of other 'features', such as seeing what actual string is written inside the R&D mode CAL area (useful for understanding how the R&D mode flags are stored if like me, you didn't originally know that cal-tool provides the same ability).

-w
I also added the ability to write a string inside the R&D Mode CAL area manually, using the -w flag. This isn't really coded to check against 'stupid' input, so if you typo something like "[rdmod or whatever you named the binary after compiling] -w -w "so much text", it will write "-w" to the R&D CAL area, not "so much text", because it will see the first "-w", expect the next argument to be the string it needs to write, and thus not process the second -w as an actual command. Then since "so much text" doesn't actually match any of the commands, it'll ignore that argument completely.

This isn't recommended by me for normal use by ANY means, but I thought some power users and people who wanted to mess around while slightly risking the bricking or unpredictable behavior of their N900 could benefit from such a feature - mainly because, if tested enough and proven safe, power users could use this tool for their own scripts to store custom R&D mode flags inside the R&D CAL area - note, however, that this too is untested - the default R&D mode flags, if all are turned on, take up 117 characters including the C string terminating Null byte. Anything bigger might cause undesirable behavior (although I have occasionally accidentally written 119 character into the CAL area, and this did not seem to break anything, but it's always slightly possible it managed to write garbage over something supposedly important - though my N900 is working fine so far).

At the same time, if/when the maximum size limit is found, the code can be updated to include a maximum character limit.

Other warnings
-s will write flags in even if R&D mode isn't on ("master" isn't already written in) - I suspect this to be safe and I THINK it'll just be ignored when booting, but I haven't rigorously tested, so when you're turning on R&D mode AND setting some flags at once, use "-e" followed by "-s". (-s followed by -e will simply make -e cancel out -s; I'll modify the behavior one way or the other accordingly when I have more time to see if R&D mode flags can be left set without the master flag being set as I suspect or not).

Everything is executed in order, so you can write -e and -d in the same command, and R&D mode will be enabled or disabled according to whatever was last. Same with -s and -c. This also means that -e, -d, and -w completely override whatever preceded them. There's no logical reason to combine commands in a way that this would arise, ever, but it's always possible someone will, so I wanted them to understand the behavior.