Infinityis

2007-01-06T04:55:00.000-06:00

New Google Notes Feature: Automated Email Backups

Google Notes is now capable of performing automatic email backups; you can receive daily, weekly, or monthly emails containing snapshots of your notepad. This is useful in general for keeping archived copies of your notes, and in the event that we have any downtime, you can at least have a recent backup copy of your notes.

To activate automatic backups for your notepad(s), we ask that you donate $1/month or $10/year to support Google Notes. By donating (via the button in the column to the right), you help ensure that we can continue to provide consistent, long-term service.

If you are unable to (or choose not to) donate, you can still create backups manually by clicking the "backup to email" button for your notepad. However, the automatic backup option exists to ensure consistent archiving with no added effort.

To receive automatic backups, click the "backup to email" button for your notepad. From your email account, reply to the backup email. In the reply, please include your email address, category name (in quotes in the first line of the backup email), and the desired frequency of backup (daily, weekly, or monthly).

One last tip for anyone who uses email backups, it may be useful to add a filter to prevent them from filling up your inbox. In Gmail, to automatically filter your notes, take the following steps:

Near the search box at the top of Gmail is a link for "add filter". Click it and a small form should display. Fill in the "From:" field with "david.hoelscher+googlenotes@gmail.com" and fill in the "Subject:" field with "Google Notes Snapshot". Run a test search to make sure this picks up the right emails, and click next step.

On the next form, you will probably want to check "Skip the Inbox (Archive it)" and "Apply the label". You may want to create a new label like "Google Notes" so you can quickly pull up a list of your note backups. Click "Create Filter" and you're all set up and ready.

2007-01-02T00:24:00.000-06:00

New Feature for Google Notes: Backup to Email

We're barely a day in, and 2007 already looks to be a good year. Fresh on the heels of the "full screen" feature, I've added a "backup to email" feature, which allows you to send a backup copy of your notes to your inbox. It's a step towards my goal of having automated backups, wherein your notepad contents are backed up to your email automatically on a daily/weekly/monthly basis, if desired. This not only lets you easily save your notes, but also lets you search through older copies of your notepad data.

If your notes aren't backing up properly, make sure you're checking the right email account. The notes are sent to the email address provided when you first started using the service. You can review this information by clicking "edit" in the top, right corner of your Google Notes. If you used some fake email address (which is perfectly understandable), then your notes will be sent to that fake email address. If you want to change the email address for your notes, you can either:

1) copy your notes, edit the email address, and paste the notes in the new notepad window

OR

2) use the buttons on the right side of this blog to add a new notepad (maybe you want to try a larger one?), fill in your email, etc., and copy over your old notepad contents.

If everything checks out and you still aren't getting your backups, feel free to let me know via email or a comment here. I've done my best to minimize the possibility of the email being marked as spam (by sending as plaintext instead of html, etc.), but we may have to feel our way around a bit before the system is flying right. Also, please let me know if you have problems with receiving multiple backups...that is, "click once, backup twice". I may enforce a minimum one hour (or so) spacing between backups to prevent accidental overload.

2006-12-16T03:08:00.000-06:00

New Google Notes feature: Full Screen

Full Screen for Google Homepage Users
Refresh for Google Desktop Users
Public Notepad in your blog or on your website
Upcoming features

For those of you who have been able to put up with our recent server outages, you now have your reward (or at least a reward). The Full Screen button gives you a lot more room for your note-taking and note reviewing.

If you use Google Homepage, this button will expand your notepad to take up the full browser window area. If you're like me and make heavy use of your notepad, this will come in very handy, freeing you of the (previously tiny) window constraints.

If you use Google Desktop, this button will open a browser window to show your current notepad (if, for example, it has been changed via Google Homepage). I've had many requests for a refresh button, and I'm still trying to see if I can get it to actually refresh within Google Desktop.

I've also corrected a recent inadequacy with Google Notes: one of the unfortunate side effects of the recent outage was that the note save frequency was changed from every 1 second to every 10 seconds. This meant that you could potentially "lose" the last few characters you were typing if you closed your window too quickly. I've put in a nice fix for this--the note saving frequency is now every 5 seconds (to reduce server load), BUT if you close the window or navigate away from the page, it'll go ahead and save it anyway.

I mentioned before that a few of you have used Google Notes for a sort of "shoutbox" or "public notepad" on your blog. The new button makes this easy for anyone to implement, so here's how you can do it:

First, add a new notepad to your Google Homepage and give it an email & category combo (the buttons to the column on the right of this blog will add a notepad for you). Click save and then click the new button to expand the notepad to full-screen. Copy the URL at the top of your browser. (You can even bookmark it if you want to jump directly to your notepad in the future.)
In your blog/website template, set up an iframe pointing to that URL. You can paste the following code in your template and it should work (if you replace "foo.html" with the URL you copied in the first step):

<IFRAME SRC="foo.html" WIDTH=200 HEIGHT=300></IFRAME>

One last thing to mention about the new Full Screen button...after you click it you may notice "up_wordcount=" in the URL. If you change this to "up_wordcount=1", you can enable a rudimentary word counting feature.

The next thing I'm working on is a "send notepad contents to email" button, so you can look for that in the future.

2006-11-30T18:04:00.000-06:00

Too much of a good thing...

is not a good thing. Don't get me wrong, good things can be good, but there must be some balance. Google Notes has been a bit unbalanced recently.

My brother Richard (who works part time for the local hosting company that allows Google Notes to exist) has brought Google Notes back online after about a week of downtime. Here's what he had to say about it:

"Everything is back online now. Technically, after replacing the server that Google Notes wound up killing, we had to find something better than mod-fastcgi for keeping the app online in between http requests. mod-fcgid works great for reqular loads and shared sites, not so much when there are 100 requests per minute. We deployed on mod-scgi instead, and it seems to be working fine so far."

Roughly translated: we had too many users and too little server.

I would personally like to apologize for the disruption in service--it was frustrating for many of you and myself alike. With that in mind, I've added a donation button for Google Notes. For starters I've just set up a Paypal thing, but if someone wants to use a different service, just let me know and I'll try to set it up as well.

There will be two main places the money will go: helping to cover the cost of hosting the service (and replacing servers we kill), and future development for Google Notes.

Two new features that I hope to have in place soon:

A combination refresh/full screen button. This will (finally) allow refreshes for those who want to sync up their notepads on Google Desktop--I've received several requests in this respect. Also, for those who use it on Google Homepage, it will expand the window to take up the entire screen. This is perfect for those of us who pack in a lot of information into their notepad.

The second feature is a "backup to email" button, which sends a copy of your notes to the email address you provided. During this last week of downtime, one of the Google Notes users suggested this and it seems (by far) the best way of handling user-accessible backups. I had considered "file->save" and cookie-based approaches, but this one is elegant and can even be automated for monthly/weekly/daily backups.

So there you have it...that's the plan. Feel free to donate as little or as much as you want (suggested amount: $2/month or $10/year). We'll still try to keep it online and keep improving it regardless of the donation amounts, but at the hosting company Google Notes has the unfortunate reputation of being the lowest income generator and highest traffic generator. Business-wise, it's hard to justify.

Also, I have considered (as an alternative) adding a small row of context-sensitive Adsense below the notes. Aside from taking up valuable real-estate, this detracts from the simplicity and cleanliness of Google Notes so I'm personally not in favor of it. If anyone has any comments regarding this (for/against), I'd like to hear them.

2006-08-28T05:09:00.000-05:00

SVG Maps

For the last few months, I've kept the idea of SVG maps accessible from an internet browser in the back of my mind. The reason isn't so much for the technological challenges involved, but for a practical effect: I've been working predominantly from a 56k modem (which is maxes out at about 44k) dial-up internet connection. The major map tools out there (Google, Yahoo, and Mapquest) are intended for high speed connections. They reload new data for every zoom in/out, and it takes a full minute for maps to load, which is painful when you want to quickly check directions before hitting the road.

Using a scalable vector graphic (SVG) map would fix this pretty well. Most map features (except satellite images) are represented by lines or borders. These lines and borders can be compactly represented as a series of coordinates, which is precisely how SVG is done. SVG uses textual data in an XML format to draw all sorts of basic shapes, complete with fills, transparency, and a variety of special effects.

By leveraging SVG for online maps, we get several immediate results:
- Faster to load. Map feature coordinates would be sent to a set of client-side JavaScript functions which translate these into the appropriate SVG elements. This would require significantly less data transfer than the usual image files that are used.
- Faster to zoom. When you zoom in/out on an image based map, entirely new images must be fetched from the server. However, with SVG, only the "new" map feature coordinates must be fetched, as previous data can be reused from a different zoom level

One downside of this approach is that to use SVG, you need a newer browser that supports XHTML and SVG. Mozilla Firefox does this, and IE will do this if you use an Adobe plugin. On the flipside, this leads to another positive:
- SVG maps are ideal for cell phones and other embedded applications. One of the driving factors of using XHTML is that is simplifies a lot of things browser-side. By requiring XML well-formedness and other constraints, internet browsers get simpler and more easily integrated into things like cell phones. If we assume that cell phones of the future contain XHTML browsers, and that cell phone data transfer rates are always lower than that of a hardwired connection, it makes perfect sense to use SVG for map data (as opposed to using images, which are worse in terms of data size, flexibility, and reusability)

As for me, I've been playing around a bit with SVG and fleshing out a possible implementation strategy. The guys at carto.net seem to have a good head start on SVG mapping, but I don't see anything (so far) that seems to focus on creating a fully-featured map that offers things like driving directions, business searches, and so on. They do however, have some samples that show how to use JavaScript to interact with SVG, talk about browser compatibility, and they have some neat demos set up.

For the most part, I've set up an rxml file in Ruby on Rails using XmlBuilder that includes some SVG data. This way, I can easily integrate information from a database, minimizing the number of unnecessary headaches. Using the .xml extension, however, means that while it works in Firefox, it doesn't work in IE, because IE simply displays the content of XML files (regardless of the doctype). If I could get RoR to output files with the .xhtml extension (and no, .rxhtml is not recognized), there would be no issue, but I haven't gotten far enough or deep enough yet to make that happen.

Here's how I would implement my proposed map system:
- Create a database of all possible map elements, along with their coordinates. For long elements, like the Texas border, I'd break them down into smaller elements.
- Assign each element a unique ID. Order the IDs based on how "localized" a feature is. Basically, SVG documents are displayed so that the first element listed is painted on the screen first, so any subsequent elements occurring in the same area are painted over top of it. The last SVG element is the last to be drawn, so it will not be overwritten. By ordering the element IDs properly, we can establish the "drawing order" so we don't need to worry about a lake being drawn on top of a bridge.
- When the user accesses the map, the center screen coordinates and the zoom level is sent to the server. The server processes this information and sends the relevant data (to be displayed + some beyond visual extent) to the client. This information includes coordinates, feature type (interstate, country road, railroad, lake, state border, etc.), feature name, and the range of zoomlevels for which it should be displayed.
- The client receives this information and processes it (via JavaScript) into SVG data and stores the original XML. When the SVG data is added, it is added in order of unique ID for reasons mentioned above.
- During panning, zooming, or re-centering, a request is sent to the server containing the new coordinates and zoom level, along with the list of unique IDs already existing client-side. The server sends any new data that will need to be redisplayed on the map.
- The client processes the new data and the stored data and paints the displayable elements (that fit into the window and meet the withinzoomlevel requirements).
- Rinse, repeat, ???, profit

Note that business and address searches can also be handled dynamically in a similar manner.

One minor caveat to this whole thing is data accessibility. In my limited search for good map data to use, I've come up relatively empty-handed. I suspect that it costs quite a bit to get good map data, and I'm not at a point where I can pony up the money for something like that. I'm open to suggestions if anyone knows where this sort of information is available to the general public. Even if data were used from a vendor, it is possible that lat&long coordinates could be extracted from the SVG source, so there may be some needs for code obfuscation, which (ironically enough) is something strange enough that I haven't really looked into it before. befor

2006-06-29T21:40:00.000-05:00

XPCOM Lisp (or XPCommon Lisp or XPLisp or Mozillisp or Mozilisp)

For over a year now I've had this idea of a lisp interpreter embedded in Firefox (or more generically Mozilla). It seems the more I think about it, the better it might be.

Presently, Mozilla uses Javascript as the "glue" between XPCOM components, XUL, and whatnot. Why couldn't Lisp (or some variant) be used as well? You could send some Lisp code to an XPCOM component and it interprets it for you, with access to other XPCOM objects. AutoCAD has AutoLisp...I would have a hard time imagining that a Mozilla equivalent would be completely useless.

To compound things even more, I found another reason why it might be good: the GUI. Lisp seems to be stuck (for the most part) in the text-based world...except for HTML applications and a few others. However, a Lisp XPCOM object could take advantage of XUL for buttons & widgets, which is handled quite nicely in a cross-platform manner. Indeed, with an XPCOM Lisp interpreter, the whole thing is cross platform...no porting necessary. One could even set up a Mozilla-based SLIME & Emacs editor (has this been done?), so the Lisp code creates Mozilla-based applications from a Mozilla-based editor & interpreter.

From what I know, it seems like there are two approaches to this method:
(1) Create a replacement for XPConnect that interfaces Lisp directly to XPCOM objects.
(2) Use the exisiting Javascript system to communicate with a Lisp XPCOM object by passing strings which contain Lisp functions.

I'm thinking (2) is a much better/simpler/more elegant solution, although there may be some limitations that I'm not aware of.

Are there any showstoppers in the idea, or is it just complicated/hard?

2006-04-30T22:07:00.000-05:00

Why Paul Graham won't release Arc

Paul Graham recently posted some startup lessons he spoke about at the 2006 Startup School. In his writeup, he mentioned:

Perhaps the most important reason to release early, though, is that it makes you work harder. When you're working on something that isn't released, problems are intriguing. In something that's out there, problems are alarming. There is a lot more urgency once you release. And I think that's precisely why people put it off. They know they're going to work a lot harder once they do. [2]

If we look at the footnote, we find:

[2] I know this is why I haven't released Arc. The moment I do, I'll have people nagging me for features.

This tells us a few things right off the bat. First, Arc probably won't be open source. If it were open source, then it could be assumed that Paul's massive following of hackers would jump on it, fix up the rough spots within the first week or two, and proclaim it to be the next best thing since Common Lisp. If it were closed source, then yes indeed, there will be an equally large push to find bugs, during which, there might be some questions raised regarding Paul's hacker status and (in)ability to extinguish bugs quickly. He'd be right to harbor some concerns.

The next logical question is, why closed source? If he releases a buggy product built around well-designed concepts or constructs, then it probably won't take too much effort to implement a similar product in an open-source manner. Open-source products generally arise because their closed-source counterparts are (1) too expensive, (2) too buggy, or (3) both. We can certainly conclude that Arc v1.0 will be buggy, and it's anyone's guess as to whether it comes with a price tag. The interesting thing about open-source is that any price tag is too much.

There's also the question of competition. With the likes of Ruby on Rails in the neighborhood, I too would be concerned about delivering a competitive web-developing product. Even at that, Y-combinator is funding would-be Arc-competitors like Infogami.

Regarding the open-source question, the easiest defense is probably something along the lines of "We're hosting a server that will run (insert name here), so we don't need to release the source code". Certainly, this formula works well for Google and the Y-combinator startups. Steve Pavlina has written extensively about finding your purpose, and for Google and the startups, they have a purpose for keeping a tight lid on their source code. They want to keep their secrets to themselves, and they work hard to make sure their secrets are worthwhile.

What would be the purpose in keeping secret the implementation of Arc? I think it's clear that Paul won't enter "desperate startup mode" and code 12 hours/day to stay on top--not now and not ever again. Thus, any startup-type business model he builds will lack the key element of determination. We can scratch off any plans for making money (read: creating wealth) startup style.

Why can't the implementation of Arc be open-sourced? I don't know. There's nothing magical about server-side code that prevents it from being open sourced; heck, Apache is ALL server-side code, and it's used practically everywhere.

What is the purpose for creating Arc? If it is to enable hackers everywhere to use more powerful development tools, then there is a chance of it catching on if it's open source. If it is to help Paul Graham make more money, then it may never see the light of day.

Paul, if you read this, I hope you're feeling a little deva vu from when you made your "Ask wealthy entrepreneurs for money, but don't ask me." This time, however it's more along the lines of "Release software early, but don't ask me to release mine." Instead of nagging you in detail about it, I'll just say: number one!

2006-04-24T20:18:00.000-05:00

Google Notes: New Print Button and Font Size Selection

In response to a few user requests, I've added the ability to resize the font and print the notepad contents. Also, I'm updating my links to use the new "Add to Google" buttons. Don't forget, in addition to decreasing your font size, you can fit more notes in by selecting a larger notepad size. The default one hosted by Google is the medium notepad.

Small Notepad
Medium Notepad
Large Notepad

Finally, I'd like to just say thanks to everyone using Google Notes. It's been fun so far, and I've learned quite a bit in the process of building and maintaining it. It's rewarding to know that so many people are benefitting from it. In fact, the number of accounts (and regular users, in general) has doubled within the last month. I don't know if it's word of mouth, Google making it easier to add modules, or even a "halo effect" from Google Calendar being released this month, but somehow there rate of new users is continuing to increase. Below is a graph showing the lastest usage statistics.

2006-03-31T15:47:00.000-06:00

Usage data for Google Notes

I must say, I'm pretty pleased with how well Google Notes is catching on. The graph below should give you a feel for how well it's doing.
Basically, the yellow dots (which look like a line) represent each new "account" created, where an account is a unique email & keyword combination. The blue dots represent the last time each account was updated.

In this three-month display (since Google Notes was launched) we can make some observations. First, in mid-February, Google added the module to their website list of 3rd party modules (www.google.com/ig/directory), and that's when it really started to take off.

Next, there are over 5000 accounts, and about 20% of those are actively used. That is, about 4 in 5 users try the module and either (1) change one of their settings and never use the created account, or (2) remove the module either immediately or within a day or two.

We can also see two other groups of users, which I've marked off with faint green lines: There appears to be a set of users that make almost daily use of their notepad modules, and another set that uses their modules every few days to a week. As for myself, right now I probably fall into the latter category, but I think it's cool that the first category even exists.

At any rate, I just wanted to say thanks to all my users and share some of the success with you. I don't know how to specifically define success, but to me, it's just the feeling of "That's cool!" when I look at a graph of the usage data.

2006-03-08T22:26:00.000-06:00

Update to Google Notes

Earlier, I posted information about a Google Personalized Homepage module I had created called Google Notes. Since then, it's been picked up at www.google.com/ig/directory and I've also posted it on googlemodules.com

After over a month of successful operation, I've applied the first update to the service. Used to be, when you edited your notepad, the time and date of the last update was displayed below the notepad. That time and date was given in Central Standard Time, which worked for me because I was in CST, but it wasn't so good for roughly 23/24ths of the world. As of last week, I had implemented not only time zone selection but also the ability to toggle the display of any information below the text area.

Over the weekend, my brother (who handles the server maintenance, hosting, etc.) was in the middle of moving to a new home AND being promoted at work (congratulations!), so Google Notes was understandably not a high priority item. However, now that things are calming down a bit, we've applied the new code to the server, run a few tests to make sure everything works well, and now it's available to everyone.

This update process is transparent to the user, and if you have no desire to select a timezone, that's fine...the only effect is that the timestamp text reminds you that no timezone is selected, and it defaults to Greenwich Mean Time.

Thanks again to everyone for making this module a great success, and please do let me know if you have any more suggestions. I hope you are all finding it to be as useful as I have.

2006-02-21T01:36:00.000-06:00

Giving Successful Presentations

I'm going to defend my Masters Research on X-By-Wireless (based on Drive-By-Wire -> X-By-Wire) tomorrow afternoon, and I'm making the last minute changes to my presentation. It's actually pretty drastic changes. My professor asked how many slides I had for the 45 minute presentation. I said I had 60, and he said I should try to cut it roughly in half. He also said that for the "outline" slide, do not read the outline, but rather talk straightforwardly for two minutes about what I did. From these two pieces of advice and a little insight of my own, I've figured out what is probably the best way to give presentations:

Never read what is on the slide. For every slide, have something else to talk about.

Giving a presentation is about hearing the presenter speak. If the audience is merely reading along with the presenter, it quickly becomes uninteresting. The purpose of the slides, therefore, is to give the audience something to listen to while you speak. If they lose track of what you're saying, the slides should help bring them back on track. The slides should support the presenter, and not the other way around.

This also has the added benefit of increasing eye contact, because the presenter isn't constantly reading off the screen, and it gives the impression of a more confident speaker.

As for my presentation tomorrow, I'm finding it relatively easy to cut the presentation in half...for every two slides, I pick one to display while the other is conveyed verbally without being displayed.

2006-02-17T15:37:00.000-06:00

Online OCR

I've tried doing some OCR stuff and lately it all seems lower quality than what I would expect. Maybe I'm thinking too highly of the status quo, but it seems like it would be possible to add more intelligence to OCR tools. For example, it would be nice if they could automatically determine that the black area surrounding a page is not text. Basically, I'd like a top-down approach to scanning. If I can make sense of it, I'd like the computer to make sense of it. Start by straightening the page and defining "areas", maybe the border here, the crease down the middle of a book there, some text over here, and an image there. The user can then confirm that these are correct, or change/redraw areas. Once they are sectioned off, then begin the actual OCR. Not just any OCR, but perhaps OWR (object word recognition) or OPR (object prase recognition). If it looks like a certain letter could be a "c" or an "e", then see which makes sense as far as spelling goes, and if necessary, see which makes sense in terms of word context. This should help eliminate the mindless choices that some OCR software requires. It's one of those things that I *know* is possible because it comes standard in office software. Easy to implement...that's another question entirely. I'd also like it to have the capability of actually converting documents into rich text (or OASIS formats) instead of just plain text.

Granted, I've only been using cheap/free tools, and higher quality tools are available. However, I'm not willing to spend cash on them. What I would prefer is that when/if I scan documents in, it's free to me, except maybe some AdSense based on the scanned text. This inplies that the OCR tool would likely be an online service. That's fine, because a server somewhere else would probably have more resources (in terms of memory, processing power, character sets) to do the job right.

I actually applied to Google to be on OCR engineer there (one of the few positions that allowed for an electrical engineering degree) but was unfortunately turned down. I know they've got a vested interest in scanning with their library/book search project, but I wonder if they might also have something in the works for public-use OCR....

2006-02-07T19:14:00.000-06:00

Cat Toys

I don't know why people insist on spending money on cat toys, like wands with feathers or cat condos. Our cat seems to have a penchant for de-feathering wands and treating our regular furniture as a makeshift kitty condo.

It's the cheap stuff that works best. Indeed, I found that our cat likes 24 AWG wire. I have a long length of it (8 feet, perhaps), and she goes crazy over it. Clearly, she finds electrical engineering fascinating. I should have suspected it when she started out on twist-ties....

2006-01-28T02:22:00.000-06:00

Google Notes (unofficial personalized homepage module)

I just finished making a little module for Google Personalized homepages (if you have/use one). Basically, it shows a small text box that you can write things down in and edit. Very simple, very straightforward. The nice thing is, if you go to a different computer or log off and log back on again, your notes will still be there. Also, there are three different notepad sizes available, so you can choose the size that’s right for you. Here’s a picture of what it looks like:

It’s completely free, and I basically did it for my needs—if you want to use it too, you’re more than welcome to do so. Here’s how to add it to your Google Personalized Homepage:

First, login to your Google homepage at www.google.com/ig. If you don’t have a Google account, let me know and I can get you one.

To add one of the new modules, click any of the button(s) below

Small Notepad
Medium Notepad
Large Notepad

Once the module appears on your homepage, fill in the blanks appropriately (for example "myname@example.com", "miscellaneous", and "Random" are filled in below). Click "save" and your notepad will be ready to go. (edit: now you can also select font size, timezone, and toggle display of "last update" time)

Note that this stuff isn't password protected, so I recommend against using a password in your keyword field or in the notepad area itself. You can get some degree of privacy by choosing an unusual keyword. Alternatively, you can create a shared notepad by sharing the email & keyword combination with your friends. Also, there is not a strict requirement that the email field contain an email address: both the email and keyword fields are just text fields and can be whatever you want.

Enjoy, and please let me know if you have any feedback...

P.S. I know that you can effectively produce the same results by sending emails to yourself. However, (1) that makes a for lot of outdated messages to sift through, (2) to edit them, you have to reply/forward the emails to yourself again, and (3) there’s simply too much effort involved if you just want to keep a simple reminder or short to-do list.

P.S.S. Just to be clear, this is not an official Google product. This is just something I made in my free time of my own interest, and is not endorsed or sponsored by Google in any way. Your data is not even stored on their servers (thus far), so if there's a problem, it's probably my fault. I'll do everything I can to protect your data from third parties and keep it continuously available to you. However, if you have some data really worth protecting, use common sense and secure it with something that requires a password. The privacy policy is here.

2006-01-25T00:34:00.000-06:00

How I lost my data and found it again
OR
How to (possibly) recover your Master Boot Record (MBR) if you overwrite it

This past week was an unusually anxious one for me. My Windows installation was giving me grief (random blackout screens, etc) so I went for the obvious solution: reinstall windows. Aside from the obvious problems, I had a poor setup anyways. I was running my OS off one hard drive, with most programs, data, and a Linux partition on my second hard drive. I wanted to put everything all on one drive.

Problem 1: I could not get the Windows installation CD to install Windows. At the time that it *should* bring up a text-based GUI with a blue background, it would just go black. I was aware that this happens sometimes with a non-Windows bootloader (I was using Grub). Confident that I knew what I was doing, I proceeded to overwrite my master boot record with something Windows liked. First, I used my Linux installation to make a backup copy of my MBR. This basically consists of using the dd command and copy over the first 512 bytes of the hard drive. The command looks like this:

dd if=/dev/hda of=~/Desktop/hdaimg bs=512 count=1

which translates into "copy 1 block (size 512 bytes) from the beginning of hard disk A into a file on the desktop called hdaimg." These 512 bytes are the MBR itself.

With a backup copy made, I copied over the MBR of my second drive, which had Windows stuff in it. The command looked a little something like this:

dd if=/dev/hdb of=/dev/hda bs=512 count=1

Reboot, reinstall windows, copy back over the original MBR, go about my merry way. That should have been it, I'd be through with my headaches. Little did I know, they were just beginning.

Reboot, reinstall windows....

Wait a sec...how come the windows installer shows only one partition on my hard drive? Where's my data and Linux partitions? Time to do some research.

After reading up about MBRs and all that jazz, I come to find out that near the end of the MBR is a set of 64 bytes called the partition table. There are 16 bytes per partition and a max of four (primary) partitions. If you want more than four partitions, you have to create an extended partition and put virtual partitions inside of it.

The root of the problem is that when I overwrote the MBR, the partition table was lost too. Without the partition table, my hard drive is just a jumbled collection of sectors. There's no organizational structure, no way to pick out files or *gasp* get to the backup copy of my original MBR.

Background information: I had no active backups. For whatever reason, all my data was on this drive and nowhere else, except a few shreds of data on a memory stick here and there. All the emails for the last 7 years, all the contact information, all the work that I'm putting together for my Master's thesis...all gone.

Needless to say, I really wanted my data back.

So, I take out an old 6 or 7 year old hard drive with 13 gigs of storage, and install Linux on it. I would use as a base to work from and see if I could possibly get my backup MBR off the drive. Fortunately, I had a plan: I looked at the Grub bootloader and noticed a short error messsage in ASCII text. If I grep the hard drive for that text, I might be able to find it. First, however, I give my brother Richard a call, as he's significantly more familiar with Linux than I am. By the time I hand up, I have an example Perl script that should work. As it's my first Perl program, I give it a try, and for some reason it doesn't work out. I didn't like the Perl script all that much anyways.

I change back to my original plan of using grep on the hard drive. The whole 160 gigabyte hard drive. I familiarize myself with the grep manpages, run a quick test on a copy of the MBR of the 13 gigabyte hard drive, and decide that it works just like I want. Here's what I had:

grep "GRUB .Geom.Hard Disk.Read. Error" /dev/hdb --binary-files=text >> ~/Desktop/hdbreal

I check for typos, and press enter. I wait.

I wait some more.

I get something to eat.

More waiting.

I give up on waiting, and decide to check back every few hours. Before bed, it's still running. I hexdump the output file occasionally to see if anything turns up, but see nothing. I wake up, same situation. If I didn't hear my hard drive running, I would wonder if it was actually doing anything. Finally, that evening, after 33 grueling hours, the command finishes, and I see the following output (look at the rear terminal screen):

Memory ehxausted. Wow. I exhausted my memory with a single grep. I check the hexdump, and there's something, but it's not what I need or want: it looks like the MBR error message but there's no partition table (which I could recognize pretty easily at this point).

Undaunted, I decide that the approach was right but my technique was lacking. I was in the dark the whole time it was processing, I ran out of memory, and if grep didn't return enough bytes, I had no way to go back to the source and grab some more. I knew about scripting in Linux, but I had never tried it, never knew how it worked. So, I looked online, checked out Linux in a Nutshell from the library, and set to work. Before long, I had a tested script that did what I needed. I looked a like this:

for ((skipamount=46280257 ; skipamount<156279257 ; ))
do
dd if=/dev/hdb of=~/Desktop/testhex bs=1024 count=1000 skip=$((skipamount+=1000))
echo "Sector" $skipamount of 156280257 echo "S" $skipamount>> ~/Desktop/hdbfound
echo >> ~/Desktop/hdbfound
grep "GRUB .Geom.Hard Disk.Read. Error" ~/Desktop/testhex --binary-files=text >> ~/Desktop/hdbfound
done
date
echo "Done"

Long story short, this script copies one thousand sectors of 1024 bytes each (i.e. about one megabyte at a time), greps the copied sector for the error message, and writes a sector number and any results from grep into an output file. It also writes the current sector number to the terminal so I can see the progress while it runs.

Note that I didn't need to start at sector 0 because I knew the back up MBR was in the last half of the drive. I did a few test runs and found out that it took about 0.25 seconds per megabyte. At that rate, it should finish in about 8 hours, which is good because it was bedtime.

I sleep, wake up, and there's still about an hour left of processing. When it's done, I grep the output file. Finally, I see the partition table data I had lost. I apparently had another MBR backed up, so I did some figuring to determine which partition table was the most recent one based on the partition sizes. I made copies of the results, and handwrote them too just to be safe. Then, I use dd to copy over newfound backup MBR (and its partition table) to sector 0 of the hard drive.

Reboot. Run Knoppix. The hard drives are there, on the desktop. I click to open one, and breathe a big sigh of relieve when I see my data.

Reboot back into Linux (on the 13 gig hard drive). Copy all my important data onto that hard drive--I wanted a backup ASAP.

With the backup securely in place, I'm now back at square one. I have my data, but I can't reinstall Windows because there's a Grub bootloader. With my newfound MBR knowledge, I use dd to piece together a new MBR that is windows-based yet has the right partition table.

Still won't run the windows install CD. I take out the entry for the Linux extended partition, and it finally works--Windows just didn't know what to do with that one partition (note: the other partitions were FAT based). I install windows, add the Linux partition back in, and put the Grub bootloader back in place. For the future, I kept a copy of the Windows MBR in case I need to reinstall again.

I tell you what, I need excuses like these to learn the details of Linux. I learned all about the MBR, partition tables, grep, Linux scripting, and a little bit of Perl just from this one panicked situation involving all my data. I think most importantly, though, I learned that an ounce of prevention (aka backups) is worth a pound of cure. I'm just glad that the cure didn't involve amputation of data.

2005-12-22T00:48:00.000-06:00

I just have a quick addition to my previous post "Lisp from the ground up".

I think one of the defining features of a Lisp processor is not so much consing or any specific command, but rather the general ability to perform operations on the program code. If a strength of Lisp is the ability to operate on the parse tree of the program, then there will need to be instructions that can change the underlying code of a program.

From my MIPS microprocessor class, I recall that there are two sections of memory for a processor: the code memory and the data memory. The code memory contains instructions (pointed to by an instruction pointer), and these instructions operate on the data and the registers. For example, you can read a certain piece or data, or you can make the instruction pointer register jump to a new address. However, there were no facilities (that I recall) to change the instruction code itself. I would consider this to be a central ability of a "Lisp processor".

Further addtion:
To put into perspective how powerful this could be, let's make an analogy to DNA. Right now, any genetic mutations we experience are a result of random, low-probability events. Imagine what might happen if (at will) we could reprogram our DNA. It could be very dangerous---we could wind up changing an individual into a new species, or worse, killing ourselves in the process. Or, we could just change our hair color. Regardless, we would have more power to make changes. In a silly way, I imagine being more like a shapeshifter/changeling from Star Trek. If we do this with processors, we can probably pack a lot more punch per transistor.

2005-12-20T15:31:00.000-06:00

Lisp from the ground up

I sometimes try to think of how a Lisp system built from the ground up might look. It's a interesting exercise, because it mostly is a matter of scope. Plus, when you compare it to "modern" computer systems, there's a big leap between low-level systems and the high level we interact with. People expect graphical user interfaces, soundcards that work, mice, hard disk interactions, etc.

We could start at the lowest possible level: chip architecture. Can a chip be designed so that it's optimized for Lisp from the get-go? Would it look any different than the chips we have now? I'm talking about things below assembly language here--logic gates put into order to build an arithmetic logic unit (ALU) are one of the basic components of a processor. Can we/have we built similar modules that are meant for basic Lisp operations, like consing? We could have new instructions that don't make much sense for someone writing a C compiler, but would be heavily used for a Lisp compiler. C compilers were (presumably) designed around available assembly capabilities, which was in turn designed around machine language features and/or processor instruction sets. But what were processor instruction sets based on? Perhaps basic operations we were familiar with, like adding and subtracting. Perhaps also features requested from higher level languages, like C. Something akin to "It'd be great if we could have an instruction that did X". I'd suppose that majority of instructions nowadays are of the latter type.

But what if Lisp were the primary language, and chip designers heard "It'd be great if we could have something that consed for us." Would we have different instructions sets? My intuition tells me yes.

Now, the chip would probably still interact with peripheral chips (in hard drives, video cards, etc.) that are customized for that purpose, and which may be programmed in C or assembly, but because all their interactions are (usually) on a low level (bits and flags), there should be no issue with those interactions.

To do something like this would probably require starting with FPGAs and some Verilog/VHDL, and result in a custom-designed chip.

Next, we could define the ground as "any existing processor". That's fine and dandy, except there's a large number of processors and they all do things differently, optimize differently, and have unique instruction sets. The fastest way to do task X on a MIPS instruction set would be different from the fastest way to do it for an x86-based set. So, a Lisp compiler, to be truly effective, would need to be customized for each processor--indeed Intel releases their C compilers which happen to produce the most optimized code for an Intel chip. Try it on a different chip or use a different compiler and it's not so fast or compact. That sounds like a lot of work to maintain; however, it is do-able. Note that Movitz plans to do exactly this, but only for an x86 architecture.

Something I imagine trying for fun would be to make a Lisp system that runs on a smaller chip, like a PIC microcontroller, or an ARM-based chip. PIC would be nice for hobbyists and electronics programmers, but ARM would be nice for PDAs. Maybe it'd be best to identify their common instructions and hit them both. I, for one, would love to convert my Zaurus to a Lisp-based OS

After this, it gets fuzzy for me...I imagine a Lisp kernel, a Lisp OS, with a Lisp compiler and Lisp applications. But somewhere in there, we put in a GUI, making the leap to user-friendliness. How to do that while still maintaining Lisp-ishness, I'm still figuring it out. Maybe something akin to the way GNU/Linux distributions have a GUI yet you can still access the terminal (which I would equate to the Lisp toplevel). We'd also have many standard protocols in there, like USB and TCP/IP, communicating just as effectively as they do now with their C implementations.

I'm also still thinking about the namespaces and toplevel issues. As I'm still learning Lisp, I may not know about it, but there would need to be some way to keep everything from being exposed at the toplevel. Maybe something akin to a directory structure (or nested namespaces), much like how in the terminal, you have to be in a particular directory to run a program. Without this, the function and variable names would become unnecessarily long.

More thinking to go, feel free to add comments if you have thoughts of your own...

2005-12-15T22:04:00.000-06:00

I've decided to give the Dvorak keyboard layout a try. This is my first post fully typed with Dvorak, so it probably won't be a long one. :) Mostly, I got the first round of practice with http://gigliwood.com/abcd/abcd.html

Also, I can't wait for a decent e-book reader to come out, using e-ink technology. I'd make one myself, but I don't have the finances for it. Donation, anyone?

2005-12-12T11:25:00.000-06:00

Web 3.0 beta

In my last post, I mentioned how noticing patterns can give you an edge and tell you a thing or two about the future. I'll make use of that today in predicting Web 3.0. But this isn't just any Web 3.0, this is Web 3.0 beta.

I've seen a few other people already referring to Web 3.0, but they're missing the "beta" part. Web 2.0 will come and go, and people will sigh as they say "still no flying cars". There was much hope and promise for Web 2.0, but it never materialized. Why?

And thus, someone from O'Reilly or Google or even a lowly unknown blogger will utter "Web 3.0 beta," and everyone will see the error of their ways. Of course web 2.0 didn't work! There was never any beta testing. Google beta tests for so long, we forget it's beta, they are easily forgiven for their mistakes, and they have longer to work out the kinks in the system. Alternatively, the name could signify that Google IS the internet, and they can brand it however they want. And since they know it's not a completed internet (the world's information isn't fully orgainzed yet), it will be released as Web 3.0 beta, and it will stay there indefinitely.

What can we look forward to with Web 3.0 beta?

In addition to Google Talk, there'll be Google Listen, which will listen to whatever you have to say and respond (verbally) insightfully, throwing in barely perceptible sales pitches tailored to your discussion.

Not only will there be Froogle, but there will be GoogleBucks, which will replace US and other currencies as we know it. It will bring forth a world-wide currency system, so your hard earned dollar here buys the same cup of coffe in Paris as you'd get in Beijing.

There will be newer and better ways of writing software. For the first time, you'll have a web-centric programming language with the power of Lisp, a fully-functional visual & verbal development environment, a thorough set of libraries that are intuitive and easy to use, it will be fast (to write and to run), and it will be a natural fit for both web-based and single computer programming.

Ladies and gentlemen, these are just a few of the features we can look forward to in Web 3.0 beta. Whether that's 5 years away or 100 years away, who knows? But that's the beauty of beta...

2005-12-07T01:39:00.000-06:00

Probability

I recently read the free book "God's Debris" by Scott Adams, and I must admit, it presented some interesting viewpoints. As Scott himself admits, there are holes to it, but some of the basic axioms presented hold some truth.

One that I found intriguing was the notion that probability ruled all. That if you flip a coin long enough, it WILL land heads up about 50% of the time. No one can change that, no one can beat that, it is immutable.

Suppose someone wanted to build a time machine, not so much for traveling through time, but rather to simply see into the future. Could it be done?

I think the more important question is, would it need to be done? Since we don't have the tools to predict the future, probability is our closest friend. We know that certain events can occur (on the average) with certain regularity. For example, 99.9% of the time that I tie my left shoelaces, I'll also tie my right shoelaces. Sometimes I might not tie the right shoelaces. If someone were to bet on my shoelaces, there is a risk that I could tie only the left shoelace.

Is it an acceptable risk to say that I will tie my right shoelace if I tie my left? I'd say yes. You won't be wrong often.

If we notice patterns, and we identify probabilities for future events, we can essentially see into the future. Not a specific future, but an averaged future.

Pattern recognition is probably one of the most valuable skills someone can have. If the right elements are being observed, and a pattern can be conjured up, the probabilities associated with that pattern can provide an edge that no one else possesses.

We can see this all around us. Patent lawyers make a living off submarine patents (as in, patent something, wait for someone to implement/invent it, then force them to licence your patent if they want to take it to market: check out the Blackberry or Eolas fiasco for examples). Casinos thrive off of the edge they create--even though their customers are well aware of the tilted scales. They may not know the outcome of each poker hand or pull of a slot machine, but they know that overall, they're going to win. Venture capitalists, too, know that the "next big thing" has to come from somewhere, and that losing 10 and winning 1 happens sometimes. But when that 1 wins big, that takes care of the 10 losses.

Losses happen. We can't change that. All we can do is find an edge and ride it long enough for probability to play out. That means (1) you must be able to identify patterns/edges and (2) they must exist in a long enough time frame.

So do we need a time machine that shows the future? I'd say no. All we need is a machine that can make good enough observations of the past, find good enough patterns, and produce probabilities for future events. Factor in learning to identify changing trends, and you're close enough to a time machine--who needs the real thing?

2005-12-05T23:34:00.000-06:00

I got back from my last company visit last week, and boy oh boy, it's cold in Michigan. I seem to have brought back a sore throat with me, but I can also boast a 3-for-3 success rate in that if I have a site visit with a company, they end up wanting to hire me.

In other news, I was reading the Dilbert blog tonight, and there was a post on how to tell if a movie is good (dilbertblog.typepad.com). Scott Adams uses the "rave reviews" source as a determining factor, but I've found my own way to tell if a movie is bad. Toothy-smile count.

If you look at the box cover, the number of people with big smiles is directly proportional to how bad the movie is. I think it has something to do with the basis of the movie being one of a "feel good" nature. "Little Women" from December 1994 is a perfect example of this: there are 5 beaming faces on the front cover. If we look instead at the movie "First Knight" or "Pirates of the Caribbean: The Curse of the Black Pearl", how many bright smiles do you see looking at you? Precisely zero.

I'm sure that exceptions about, and that I haven't done enough research to broadly justify my claims, but given the five minutes I spent looking into it, I believe it warrants consideration as anecdotal evidence.

And that's good enough for me. :|

(By the way, you can tell this was a good blog post, because the "smiley" at the end wasn't smiling.)

2005-11-28T22:24:00.000-06:00

Busy times...

In between visiting companies and working on my thesis, I've had to do some thinking about the future. I'm at an exciting point in my life where I need to choose what direction I want to take, and the choice I make will (likely) have a long-term impact on my life.

I've gotten a very nice job offer from a defense contractor, and it was a good $5k/year higher than what I was shooting for. I'd have to get secret or top secret security clearances, and the job seems to offer long term stability--most employees have been there a good 20 to 25+ years, and the full range of benefits are offered.

I've interviewed with another defense contractor--no offer yet, but it'd likely be lower. I'll be interviewing with a Big 3 automaker later this week, and they'll probably have a pretty good offer.

Since I have a background in power electronics and motor drives, with a particular interest in hybrid electric vehicles, the work I'd be doing would be no surprise at any of those companies...the first wants me to do power electronics specifically, the second wants me to do either power electronics or military hybrid vehicles, and the third would want me to work on hybrid vehicles.

Maybe I should blame Paul Graham, but I don't know that I want to take any of those opportunities. What I want to do is work for a startup and work on cool technologies with minimal overhead. (Just for reference, government defense contractors and big 3 automakers have significant overhead/red tape/politics/etc.) Normally, my electrical engineering background would make this difficult--motor drives don't come cheap. Making a good motor from scratch is much more expensive than writing code from scratch. Plus, because it's a physical object, scaling means more than buying more/bigger servers.

For what it's worth, I could combine both my interest in CS and EE by programming my latest idea: LiSpice. Basically a combination of Lisp and Spice (for electrical circuit modeling). Spice uses text input, and it can get tedious to write, so I'm thinking maybe I can save keystrokes by using Lisp to write the Spice text files. But I digress.

Another avenue has been made available to me, and I think I'm going to take it. My professor (who wants me to stay on for my PhD) is in the process of setting a up a startup company. The startup will be focused on one of my primary EE interests, and I would have some share in the ownership of the company (although it's too early to know how much at this point). At first the pay won't be as good as my other offer(s), but that can certainly change with time. Also, I can really only see six months out...the startup may or may not happen, so there's certainly a lot of risk to be had.

(High paying job + stability) vs. (potentially rewarding startup + lower initial pay + less job security)

Which wins? I'd don't know 100% for sure yet, but I'm leaning toward the startup.

I think one of the biggest draws it has is that it's in line with my passions in life. I recently wrote down what it is that I want to do with my life:

"Develop sustainable energy solutions for the world, and make more efficient use of the energy we have available."

Long term, I'm pretty sure I want to work technology like biofuels or something else that essentially uses energy that's consistently replenished by the sun (solar, hydro, etc.) I think the "oil habit" the world has is starting to cramp our style, and supply and demand will make it easier for disruptive technologies to take over--and if I have anything to say about it, those disruptive technologies will be sustainable. (Note: I'd be ok with earth-based nuclear energy as short-term solution, but even that is technically a non-renewable resource).

I want to bring about a long-term solution to our current pesky energy problem (even if it's not a big problem yet; if we wait to long, it'll be too big of a problem).

Regarding the startup, I hope I don't shoot myself in the foot on this one. It feels wrong to pass up a higher job salary + stability for a low salary + instability, but I think I can really make a difference by going with the startup. If the startup doesn't work, I may have a difficult time finding a job--who knows if these other companies will still want me? Fortunately, even if they don't, I have some pretty strong leads at other companies. I'm sure everything will work out, one way or another.

2005-09-19T00:48:00.000-05:00

The middle ground.

I've noticed a lot of comments about Google and what they're doing next. One popular idea I've heard suggested is that they're going to try to deploy something to replace (1) Microsoft Office, or other office suites or (2) Microsoft Windows itself, or other operating systems. I won't comment on (2) because it's still a bit too far away. However, I think (1) is possible, but it'll take some work.

The problem is simple: to have an office suite (or any other program), you have to send a lot of program information. Even worse, you have to send fairly complicated information if you want interactivity. I have an exceptionally difficult time imagining someone implementing an office suite using HTML--the pages are inherently static, and you'd have to embed JavaScript to have any level of interactivity. Presumably, this is one of the nice things about Java. "Write once, run anywhere." You could write an office app in Java, send it out, and voila, problem solved. But not quite. Once someone has that office app, then they have it forever, short of deprecation and DRM-like restrictions that get in the way. What they'd want is to provide a service, so no, that isn't it either.

How about X-Windows? That's the ticket! Just host connections to their servers and run the office app from there, problem solved. But we now have new problems. Not everyone runs X-Windows...if you use Linux or Unix, no problem. If you use Windows, then you can either buy XWin32 or download and configure a XWindows program using Cygwin or something. (Note: this isn't so trivial as it sounds...I tried once, but I just gave up and installed Linux for the first time, which ended up being a much simpler solution). Now, they could release a free (cross platform?) XWindows program for people to run their office apps, but that has it's problems too...primarily that it's SOOO slow and unresponsive. I tried to run a visual ANSYS environment connected to a university computer, and the graphics refresh rate was incredibly slow.

So now we're close to a solution, but we're still not there. We need a middle ground. Something more flexible than html, but less flexible than XWindows, and won't run without an internet connection. (Yeah, that last bit might seem restrictive, but since Google's prime choice is web services, it wouldn't make sense to provide services that don't require the web.)

Let's see now...we all use html, and a web browser to view it. The computer doesn't have to transmit the shape of each letter, it just says "this letter in this font and this size" and the browser takes care of the rest. Why not have another markup language for applications, and use a different kind of browser to view things with it. The browser would be responsible for "common" features like menus & menu bars, child windows, standard icons, panes, messageboxes, and even common controls like sliders, buttons, treeviews, etc.--the basic stuff that we expect a variety of programs to have. However, the business logic is all done by the server. When the Application Browser connects to a remote application file (.raf), the browser receives enough information to populates its menus, bring up the default screen (maybe with a logon request?), and establish a working (non-static) connection with the server. When a menu is opened, it responds immediately (because menu processing happens on the local computer), but when a menu item is selected, a corresponding request is sent to the server (except for, say, menu/toolbar customization request). The processing is handled remotely, and the screen is updated appropriately. The only significant source of lag is if an image or graph is being displayed, like when someone performs panning on a map or zooms in on a chart.

The nice part of all this is that a minimum amount of information is sent to the users computer. Just a little text here and there, and the Application Browser knows how to do its thing. I must admit, one of the best things about this approach is that since the information is reduced to only its critical text-based components (except in image loading), the use of Lisp is greatly rewarded. I'm familiar with what Paul Graham did with Viaweb over or about 10 years ago--he basically used Lisp to not only serve up web pages but also allow the easy creation of said web pages. I've found that the reason he could do it was because he was working with a small, well-defined output format: HTML. It was static, simple, and no one expected much of it. This was before Ruby on Rails, flash, and (I could be wrong about this one) JavaScript. Interactive web pages were not required. Because he had a limited playing field, he could easily manipulate it as he needed using ONLY text files. Now, technically, someone could come along and write something in Lisp that outputs JavaScript and html, but seriously...using a programming language to write code for another programming language? That's kinda crazy--it's way more sane to write an programming language that produces output in a certain standard protocol. What we need is an Application Protocol. Something that allows us to express (in text) the basic interactions of software between the server and the user, without being bogged down in details (like getting the menus wide enough to accommodate all their entries).

Also, because the developer is hosting the program, access to the program is under control of developer, as well as updates and bug fixes.

Bottom line, html has outgrown its shell. These things like JavaScript are patches that emulate interactive web pages, but the bottom line is, they're still web pages. Web programs, on the other hand, seem to have a future. Unfortunately, it doesn't look like there are many (any?) protocols in place to handle this. I could be wrong...I still have to look at J2EE closer, and maybe CORBA ORBs. They might provide an answer, but I'm still learning.

2005-08-19T15:14:00.000-05:00

Cooking and Programming

I do a good bit of cooking. I'll make muffins, bacon & eggs, macaroni & cheese, pancakes, or anything like that. But the thing about most of that: it's simple. Except for bacon and eggs, I get stuff that comes in a box or bag and is easily prepared by adding milk or water, or a few other minor ingredients. But by and large, it's "open box, follow instructions".

Some people might not call that cooking. They'll call it warming up food, heating up a meal, or "just add water" meals. To some, cooking isn't cooking unless you start from scratch, follow a recipe with 20+ ingredients, and add your own seasonings. But for them, the cooking and the meal is an experience, and they will not settle for anything less than the best they can prepare.

It hit me today that programming is much the same way. Computer programmers can either use Java, assembly language, or a number of other languages in-between. With Java, all you have to do is add a little business logic and the GUI, messaging, and object model are all provided. Whereas with assembly, you start with the basic of basic functionality and build up. It would be like growing grain to feed a cow to butcher a cow to have meat for your hamburgers. Not many people would be willing to do that, and it wouldn't be worth the effort, since programming at such a low level is largely unnecessary--not much can be added to it by doing it yourself. But if you buy your meat from the butcher, you can still do everything with it you had planned to originally. I would approximate that buying from the butcher is the equivalent of using C or C++. You don't have to worry how it got there, and you have complete freedom in what to do with it. It may take a bit of effort to get a GUI set up, but it can be done, and you can do it however you like. If you buy some microwaveable chicken tenders, however, that's like using Java...unless you do something special, it'll come out the same every time with a lot less effort on your part. You pay more for it (either in $$ for chicken or processing speed for Java), but to many people, it is worth the cost because their main goal is not to find the best way to prepare chicken or find the most efficient way to program something.

Therein lies the heart of capitalism. If you do care about the best way to prepare chicken, the best GUI, or the most efficient algorithm, then become a chef or C programmer. If you think you can do it better than the average, and that people will want the way you do it, then you have a market to which you can sell your work. If you can do it better but nobody cares because what they have is "good enough", then you'll be a starving artist. So we must all choose carefully our profession--if we want to push the envelope AND make a difference, we must be able to (1)identify an area that can be improved upon, (2) strive to learn about it, and then (3) push the envelope.

2005-07-25T21:23:00.000-05:00

Topic of the day: Open Source PIC Bluetooth device

I've been interested in Bluetooth for awhile now, tried to understand it and everything like that, and I think I've done a reasonable (yet still not complete) job. From day one though, I was causing trouble. I asked the "wrong" questions, and people corrected me on them...'no, you use a host controller' instead of a microcontroller, things of that nature.

Now that I've figured out a lot of the components involved, I find myself asking some of those same questions, but this time I think I know some answers.

For example: "Why can't I just tell a PIC microcontroller to speak Bluetooth and hook up an antenna to it". The short answer is, there's no reason why not. The long answer is, it hasn't been done yet, and it would be hard to get the antenna right (because there are very exacting requirements for the antennas and most of that information is considered trade secrets).

So, next question: "If I had the right antenna, matched properly, etc., why can't I just connect it to a pin on the PIC and use to PIC to communicate with other Bluetooth Devices?" The short answer: technologically, the radio is hard to do. Bluetooth devices are courteous in the sense of Device A and tell Device B "Hey, no need to shout so loud, talk quieter", and Device B can respond by turning down it's output power. With a PIC, all we have is 0s and 1s as pin outputs, so it would be hard to reduce the radio power unless we had a separate system (note: this is why a lot of wireless devices from radio modems to Bluetooth are generally packaged away from microcontrollers).

Moving on: "Ok, so I've got a controllable radio and I've got a matched antenna. Now can I hook up a PIC and make it talk in Bluetooth?" Short answer: Maybe. Here's where things start getting fuzzy. Let's assume we can talk to the radio in 0s and 1s (which is a reasonable assumption). If we see an incoming transmission that says "turn up the broadcast power", we can use 0s and 1s to tell the radio to adjust power levels (to be in the other device's Golden Receive Power Range, as they call it in the Bluetooth spec). Anyways, so that aside, yes, when everything is a 1 or a 0, the PIC can do it. But the PIC may not be able to do it fast enough. I haven't thoroughly investigated it, but there are some things that need to be done with those 0s and 1s that are easily done with hardware, such as using Linear Feedback Shift Registers. To ask the PIC to do such low-level processing would take up clock cycles, and it could not be interrupted during this time, because Bluetooth timing stuff is VERY sensitive. All these operations (the time sensitive things that typically lend themselves to hardware implementations) are called Baseband operations. And for Baseband operations, its probably for the best (but by no means necessary) if it's implemented separately from your primary processor.

Let's do a quick summary...We need an antenna, radio, and (preferably) baseband to do ANYTHING Bluetooth. There's more we need on top of that, but we'll get to that later. First we have to take care of licensing issues. Bluetooth is trademarked, which means you can't even sell Bluetooth toothpicks without talking to the BQB (Bluetooth Qualification Body), an organization sets up by the Bluetooth SIG (Special Interest Group) with the sole purpose of distributing licenses that allow the use of the name Bluetooth. Obvious question: "Can I make something that interacts with a Bluetooth device that is not qualified through the BQB?" Yes you may, but you can't sell it, or if you do, you cannot use or have the Bluetooth trademark, logo, word, etc. anywhere in your stuff. Why might someone want to go around the BQB then? Money. I understand it costs about $10,000 to certify/qualify something as a Bluetooth Product. That ain't pocket change for Joe Electronman who tinkers PIC microcontrollers at home in his free time. Bad news: Bluetooth is only for the big boys. The American notion of making a fortune when you start with only a dollar in your pocket and an idea in your head...that doesn't happen much with Bluetooth stuff.

There is a middle ground though: Bluetooth modules. These things use standard communication protocols (so PICs can talk to them), offer a variety of features, are fully qualified (so you don't have to) with the BQB. You can put one of these babies in your product, slap the Bluetooth logo all over it, and nobody can stop you. The problem: (right now) they cost about $60 to $70. That might be alright for small experiments at home, but if you want to mass produce a cheap device, it might be a problem.

So there is a decision to be made: are we going to play serious and make something that sells like hotcakes, or are we home hobbyists? Home hobbyists currently have no recourse other than buying expensive modules. If we want to play serious, we better have some money, because here's what we can do...we can (for example) buy an antenna from gigaAnt or another provider and then go to btdesigner.com and buy radio+baseband+otherstuff single-chip solutions that provide up to RFCOMM (whatever that is). Great! These chips are only $14 each (min order of 5), and if you buy 100 of them, it's only $8 each. Now we're talking. (note: I haven't priced the antennas). These chips have 90+ pins on them, but that's ok because we can read a datasheet and do all the cool stuff we want.

Almost.

Going this route, we still need to go through the Bluetooth qualification process (yep, $10,000). Why? Well, aside from the fact that the chip and antenna could be mismatched, Bluetooth devices in general aren't controlled from the RFCOMM level. Huh?

Ok...analogy time. We know radio and baseband, but what is the Link Controller, Link Manager, L2CAP, HCI, RFCOMM, OBEX, WAP, TCS, SDP, and these Profiles I keep hearing about? They are all layers of abstraction. Think of the English language. A book is composed of chapters, chapters contain paragraphs, paragraphs contain sentences, sentences contain words, words contain letters, and letters are specially shaped globs of ink on a page (or pixels on a screen). When someone asks you about a paragraph you read, you give a nice summary of it. It wouldn't make sense to say "Well, it started with the word 'It' followed by the word 'was'." It would make even less sense to say it was the letter "I" followed by the letter "t", and so on. For that matter, it would be impossible to say the first letter on the page looked like this--at which point you motion in the air what the letter looked like on the page. Why don't we do this? Because it's unnecessary. We all know what the letter looks like, and we all know how to combine the letters into words. There is a standard way of interpreting these things.

Therein lies the beauty that is Bluetooth. Bluetooth standardizes the shapes of the letters (a.k.a wireless signaling). Bluetooth standardizes the communication on many levels, and in doing so, the ensure interoperability--that is, they make sure that if one person can read the book, everyone can read the book. All the acronyms earlier are just named levels, just like we could name the levels "letter", "word", "sentence", and "paragraph". To have a sentence, you MUST have letters and words, but you don't need a paragraph--in fact, the concept of a paragraph in a sentence doesn't even make sense.

When they say that the CSR single-chip solutions implement everything from the radio to RFCOMM, it means that it needs an antenna (a lower level) to be useful at all (without it it's like a book without any alphanumeric symbols in it), and it needs some higher up levels to perform reasonable communication with other devices. Specifically, the higher up levels it needs are profiles. Profiles are the lingua franca of Bluetooth. They are software protocols that allow for a variety of operations, from object (file) exchange, to discovering what services other Bluetooth devices offer, to allowing real-time audio broadcasts. Some profiles even use other profiles, essentially creating new layers that abstract the underlying process.

For what it's worth the RFCOMM layer can be treated like bunch of RS232 serial ports, which means you can pretty much send all the data you want over them. Note that if that's what you were planning to do with the PIC, these single-chip solutions are pretty suitable, because you can treat them like a RS232 port, which is where the notion of "wire replacement" with Bluetooth comes in. Note that because

So, we get an antenna, we get a chip, now we can write some code. We implement any profiles (and any unimplemented underlying layers/profiles) we need, and we can happily do this on a PIC or wherever. I haven't looked much at the Linux BlueZ protocol stack (protocol stack = a collection of the layers. The rules for sentence structure and paragraph structure could be considered a protocol stack), but you could probably use some of it to implement what you need. However, if you modify it, you'll have to get re-qualified with the BQB. Fortunately, the BlueZ stack has undergone qualification as-is, so that's one less thing to worry about.

Now we just assemble the antenna, chip, and (optionally) a microcontroller (note: if your program is small, you can use the on-chip microcontroller), and make sure your program operates with a profile, send it to the BQB if it needs it, and you've got a fully qualified Bluetooth device. Do keep in mind that you'll probably need a surface mount machine to assemble the chip to a board.

So why did I write all this? I think because I had so many questions that I wanted answered, and I didn't find any answers directly. The level of investment (financially) required for Bluetooth is generally enough to exclude hobbyists, but all good hobbyists insist that there MUST be a way to do what you want. And so there is. And hopefully I was clear enough that I could help someone else understand how to do it.

Another reason I wrote it was because it may be possible to have an open-source PIC device with Bluetooth, but for technical reasons it will still need something separate for the radio and antenna, unless Microchip does something cool (and maybe a bit out of their league) and produces a Bluetooth-enabled PIC microcontroller. Heck, if they could even integrate and/or have an add-on antenna, that would be awesome. I imagine if they did, they'd dominate a lot of the market, open up Bluetooth to hobbyists, and basically bring in a revolution for Bluetooth products. They'd also probably do it at the right price too--$60 to $70 for a Bluetooth module is too high, but it's just because of the technical expertise necessary to make and match an antenna, add profiles, and assemble, all for a (currently) low volume product.

2005-07-23T01:28:00.000-05:00

Two new ideas today: Googlesaurus, and Wisp.

Googlesaurus: A Firefox extension that replaces a Googlebar. Basically, the problem with Google searches is that you have to know the words you're searching for, otherwise you don't get a good search. While many of us may have excellent search skills, a large number of people put in terms that are too vauge or generic to produce good results. Googlesaurus would provide the standard Google search box, but when you perform a search, your search results are displayed word-by-word in the toolbar, each with their own dropdown list. You can click on the dropdown list for any word and see a number of synonyms (maybe even some antonyms) for the words. If you find a word that is more specific than your original one, you can click it and do a new search.

Wisp: Kinda like Lisp, but better for AI. At a fundamental level, brain cells have a number of branches that reach out to other brain cells. Lisp (list processing) is like a brain cell that has only one branch that goes out to another cell. Wisp (web processing, with the "is" part borroed from l"is"t) would have an object and a series of pointers to another object. The idea is still very fresh in my mind, so I've got a lot of hammering out and figuring out to do, but that's the gist. Heck, it may even be implementable in Lisp. Time will tell...

2005-07-20T20:25:00.000-05:00

I've been doing lots of reading lately...basically getting into the details of some programming, especially regarding the Mozilla Framework. Something I think that would be neat/good/etc is a new kind of Mozilla suite with a different purpose: to provide a range of open source cross-platform engineering tools. Things from 3D CAD systems to FEA tools to mapping tools, and maybe even some Matlab/Labview style tools. I was thinking a good name for it might be Vizilla, to emphasize the graphical nature of the program. I've been reading a lot about XPCOM, but I haven't gotten far enough yet to see exactly what needs doing. I understand Gecko does the web browsing for Mozilla products, so I'm not sure if that'll be something to replace with a more suitable graphics tool, or if it's so deeply embedded in Mozilla that it'll be hard to get out.

Anyways, in other news, I started thinking that it would be nice to run Lisp scripts in Mozilla products, kinda like Javascript, but have it set up like Autolisp where you can issue commands to do this or do that, or a whole buch of thises and thats. I did a quick search in Google for something like that, and I actually came up with something...NYC lisp is doing a Summer of Code project called Firelisp, but it was already past the deadline. So, I applied anyways, but nothing so far.

Back to reading...

2005-07-09T14:48:00.000-05:00

I've been going deeper into Java, trying to figure it all out, and the more I do, the more it looks inherently different from Lisp. Basically, I'm not so sure it's a good idea to do a direct-to-.class file Lisp. It might work, but I've got more learning to do, for sure.

In the meantime, I found a nice (actively developed) Java-based lisp interpreter called Armed Bear Common Lisp (ABCL).

Something else I've been thinking of lately is how much wasted code there is. I've heard the quote:

"Programs must be written for people to read, and only incidentally for machines to execute."

- Abelson & Sussman, SICP, preface to the first edition

But I'm not so sure I believe it. If I were to be quoted, I think it would sound better as:

"Readers (a.k.a. compilers) must be written for people to program, and only incidentally for others to make sense of."

The fundamental difference is the purpose of the program. The first implies that the first purpose is for other people to read programs. However, I can say with certainty that the large majority of people prefer to use programs rather than read source code. That said, the fundamental purpose of the program is to be used. And if it is easy for people to write programs, it is easy for people to communicate concepts and ideas to others in an easy-to-digest manner. The programs that help us do this, I classify as compilers.

Under this definition, we find that not only are C compilers and the like within the definition, but we also find word processors and email programs. Indeed, these programs help us compile our thoughts into discrete words, which we can transmit to others to widen their breadth of information.

But I digress...while it makes sense to have some readability of code, it also makes sense to have more compact code. For example, the latest download of the Fedora Core iso files takes 4 discs. I cannot possibly imagine that all of that is really necessary. I imagine that somewhere, some program wrote from scratch a necessary function which duplicated another function for another program. In fact, I imagine this happens hundreds of times, if not more. What if instead this series of statements was recognized as "common" and put into a library. Indeed, what if this were done for thousands of functions and programs. With today's availability of memory, it might be possible to significantly lower hard drive accesses, because only user data and program-specific code would need to be fetched. Also, handheld devices would benefit, as they could hold a lot more programs per megabyte.

If anyone knows of such a program, please let me know, as I'd be interested in it...

2005-06-22T16:25:00.000-05:00

Lately I've been thinking a lot about mixing Java and Lisp. The reason: Lisp is powerful stuff, but limited by it's text-based I/O. Java is powerful stuff, but limited by some of it's cumbersomeness and lack of flexibility. So, I started looking for anything that exists that might combine the two, hopefully in a way that allows Lisp to access the Java libraries and also compile directly to bytecodes to ensure interoperability on different platforms.

I found some Java-based Lisp implementations, and I found stuff that lets Lisp access the Java libraries, but I had a hard time finding something that did both. Finally, I came across Armed Bear, which seemed to be what I was looking for: it was completely Java-based, right down to the editor, could access the Java libraries, yet was at version 0.0.6 and had some major restrictions.

Then I started thinking some more about why Lisp had the limitations it did. Lisp was developed in the 1950s, and way back then, all you had was text input and output. And when the ANSI Common Lisp standard was defined (1990s) there was mostly text I/O. GUI's didn't come around until the 90s. So why would it include anything but text input and output? GUIs changed everything.

The next logical question is, why isn't there a "new" Lisp that deals with GUIs and such? Was the language to complicated to begin with? To simplistic? Is it not conducive to functional programming or interpreted programming? Can we redo functional programming so that it works with a decent GUI? And what GUI should be used--Xwindows, Windows, Java style? Standardization is an issue with that. Used to be, you had a command prompt with a flashing cursor, and no matter where you were, that command prompt would be there (usually). There was a place to input, and a place to read output.

Is Java the modern Lisp? Many would say "no", because it takes longer to code stuff. Maybe it just takes longer because people want more than just a text I/O. I think maybe this is why Paul Graham happened to be so successful--the internet spoke in text and static graphics when he was a success. As such, he could manipulate the text (html) all he wanted, and the graphics weren't going to change anyways, so it didn't matter. But even now, 10 years later, we're seeing more and more interactivity with the internet, which results in dynamic graphics, and graphics are difficult to do with text. Not impossible, mind you, but difficult.

Should it be so hard? Can it be done more easily than Java classes and inheritance and implements and such?

Let's suppose I give it a go--I start building a Lisp program that will output either Java code or bytecodes directly--it shouldn't matter which, just one will be more concise, one will be clearer. Now, the JVM seems to be based on the concept of classes, which makes perfect sense for Java--everything in Java is inside a class. Should the Lisp code be embedded in the classes, or should it supercede the classes? If it's embedded, we lose out on the "interactive" aspect of Lisp. We'd just have classes like normal Java, a bit of Lisp code to carry out stuff, and it'll all run like normal Java.

I think that's a losing proposition. What if instead, each entry in Lisp submitted a new .class file to the JVM. That ensures the "interactiveness" of Lisp, but each Lisp statement would have to be contained as a class. I don't *think* that would be too much of a problem. I'll keep looking into it and update as I learn more about the JVM, Java, and Lisp.

2005-06-15T00:01:00.000-05:00

First, thanks for stopping by.

Second, sorry I'm late. Yes, I'm late. Most everybody and their dog has a "blog" now, and I was quite content to go without one for awhile. My big reason: the word. "Blog" is such a boring word. Short for "web log", it sounded like a cheesy rendition of "The Blob" or "Blah". However, I've become more forgiving of the word lately, as it's popularity has increased and it has been replaced by even dumber words, like "Grafedia", which was listed on Slashdot today.

I suppose this is to be expected...languages are developed in accordance with their significance in society, and this is why there are hundreds of words in the English language that deal with money, sex, etc. So in such an information-rich era, we will undoubtedly see more and more words arise which deal with the generation, transmission, and presentation of a wide variety information and ideas.

Within this site, I plan to release some of my own generated thoughts and ideas along with a healthy dose of knowledge and wisdom acquired from other sources. Hopefully it'll be as much fun for you to read as it will be for me to write. If not, you're more than welcome to read my dog's blog instead.