» tagged pages
» logout

sorted by: recent | see : popular
Content Tagged with v9fs + inferno

Service Oriented File Systems

Its fun to abstract and think about web2.0 concepts in terms of Plan 9 concepts. Abstract away the gorp that makes up their implementations and think about some of the fundamental concepts. When interfaces are files, mashups are binds.

The larger picture here is about keeping this simple, language independent, and client independent. I'd like an infrastructure that I could build simple web services out of that can be viewed with a browser, or with a rich-client (like acme plug-ins). I'd like the front-end of these services to be defined in the same way the Octopus approach defines normal GUI's (except from a web-browser, the widgets are implemented by Java script). I'd like the back-end of these to be akin to big table and the chubby lock service, with a back-back-end that is simply Venti versus a database or XML.

I'll admit I'm probably naive about all of this, not having really worked on client-facing applications for some time, but I think it would be a fun thought exercise to go through and perhaps a nice refresher to actually try and implement. While part of the goal is to keep things language independent, I'll probably do things under Inferno to keep them portable for me and open up the potential for interaction with the Octopus crowd.

Of course, this all falls under the copious spare time which I have a definite deficit of -- however, I think dedicating one day a week is doable and I should be able to combine this with my constant desire to get back and look at collaborative facilities (aka warren) under Plan 9 and Inferno.

v9fs: Grave Robber's from Outerspace

GSoC 2008 Ideas

My Plan 9/Inferno summer of code ideas (otherwise known as projects I wish I had more time for) - projects should take no longer than 5 weeks to complete (including ramp up, debug, and packaging) -- conservatively projects should be completable in 3 weeks by experienced P9/Inferno developers and 2 weeks by folks unfamiliar with p9/inferno development environments.
  • mount.9P helper program for v9fs including packaging for debian and fedora -
    • hueristic for determining transport type
    • DNS support for resolving hostname <->ip address
    • ssh-based tunneling support (will require server work as well)
    • man pages
    • debian packaging
    • rpm packaging
    • (stretch) authentication support (w/p9p)
    • (stretch) authentication support (w/o p9p)
    • (stretch) integration with idmap solution
  • since this is relatively straightforward it will really need to be "super" version including support for ssh tunneling, authentication, etc.
  • idmap solution for v9fs - maps local uids/gids/error-codes to strings
    • userspace daemon to provide mapping
    • hooks in v9fs to use mapping
    • (stretch) synthetic file server approach to update mappings
  • OLPC Inferno Environment
    • fontfs - on-demand ttf solution
    • metafs - abstract metadata from underlying file system
    • (stretch) Integrate OLPC translation solution for Inferno
    • (stretch) OLPC oriented GUI toolkit, window manager, and toolbar
    • (stretch) Inferno approach to collaboration using file systems
  • Source Control Management based on Venti backend
    • single branch version tracking with associated log file
      • add new files/directories
      • commit changes
      • checkout specific version #
    • (stretch) support multiple branches
    • (stretch) support repository sync (push/pull)
    • (stretch) support three-way merge
  • wrapper for p9p vbackup to make it more user friendly
    • wrapper which tracks venti scores based on volume being backed up
    • GUI admin tool
      • which sets up venti in partition(s) or with files
      • which assists in configuring backup intervals and volumes
    • (stretch) time-traveler like GUI for navigating/searching backups
    • (stretch) support for pruning and/or merging snapshots
  • GSoCFS for managing future community involvement with GSoC
    • synethtic file system and web interface
    • posting project ideas
    • voting for projects
    • registering student interest in projects
    • project milestone tracking
    • project blogs and wikis
    • post-summer project success metrics (subjective and objective)
    • (stretch) syndication points for community monitoring
    • (stretch) potential integration with SCM
    • (stretch) potential integration with some form of chat
    • (stretch) potential integration with name space sharing
(more to come...)

v9fs: Grave Robber's from Outerspace

inferno-olpc circa 1/28/08

This week's update to inferno-olpc includes OLPC-optimized fonts and a customized logon with two extensions:
  • it binds the Inferno activity "writable" area to important areas of the Inferno name space which the user may want to write to (/usr, /services, /keydb, /tmp). This will be necessary for the next OLPC update which will start enforcing security (including only allowing writing to the activity writable area of the file system. Also bundled is a day-one scenario which will setup template directories to bind from and copying relevant files.
  • when logon can't find a user it will now prompt to create the user by copying the template /usr/inferno directory to /usr/username. This can probably be generalized back into the standard Inferno base.
I also have a "customized" toolbar which just limits the size of labels to 15 characters so you don't get the awful line wrap. This is a stopgap, a better toolbar metaphor needs to be developed for the OLPC version of Inferno. I also augmented wmsetup with a larger set of applications.

I had a larger agenda that included social-network style "profiles" and integrated security/password setup. But there were too many corner cases to test and "bullet-proof" so I decided to hold that off for later and maybe just incorporate those functions into separate applications.

I'm still a little uncertain of how I want to handle authentication and security. On one side, there could be a central "schoolserver" to setup authentication against. However, as most of the current users don't have such a server, having a pure peer-to-peer mode is probably preferable. Given that every laptop will be an auth-server then, do we setup a laptop wide authentication setup, or do we do a per user authentication setup. The later would seem to make more sense to me as the user-provided password wouldn't have to be stored anywhere and could be use to setup keyfs and the user's own authentication secrets. Other users "registering" with that user for the purpose of collaboration could then provide their own secrets in his keydb, etc. I'm not sure if that makes sense or if there is a better model I should be considering.

The file system security is another pain. Inferno users won't necessarily equal underlying system users (a common problem). This will be made worse with the new OLPC security enforcement which may have different uid's for each "instance" of Inferno (but the same "gid"). I think the right thing to do is a new version of devfs which always keeps group permissions open in the underlying fs and uses extended attributes or some other mechanism to maintain Inferno uid's and gid's for the files.

v9fs: Grave Robber's from Outerspace

freefontfs - a styxserver inferno-lab

I've decided doing the dynamic free font file system would be a more gentle introduction to using the new Inferno styxservers modules than a full-blown filterfs like what we plan for metafs.

The most appropriate example would appear to be mntgen.b (which allows for dynamic walk based creation in /n). It uses both the styxservers facilities and the navigator and has a dynamic nature similar to what I want for freefontfs.

Fonts are comprised of two types of file, the base file (..font) and subfonts (..). We only want to generate (and cache) files as they are accessed. The base available 's will be derrived from crawling a directory for ttf files or perhaps from a configuration file.

Following mntgen, it looks like it would be much more simple to keep it as a single level hierarchy (much like /n) -- this may be possible as all the necessary information is encoded in the filename. We just need to parse the single walk correctly.

One question is whether we should prepopulate the cache with a base font size for every ttf available. This seems like a reasonable idea, but probably wouldn't even be necessary in many situations -- perhaps we can leave it as a config option.

v9fs: Grave Robber's from Outerspace

metafs

Since I wrote about it in my other post, I might as well explain metafs. Basically in order to accommodate the OLPC file system security model I need to maintain uid, gid, and mode bits for Inferno's view of the underlying file system separately. This is actually a common need within hosted Inferno when you don't necessarily want to keep Inferno user's sync'd with system users. In any case, this amounts to an overlay file system, which stores away the additional metadata.

While I am tempted to keep a central database of the metadata, the most pragmatic solution would seem to either use extended attributes or a hidden file in each directory. Since extended attributes are somewhat distasteful and not available anywhere, I'm going to start with the per-directory hidden file.

In the future this same mechanism might be extendable to implement semantic file system extensions (or extended attributes for that matter) -- allowing for tagged organization and searching of the file system as well as alternative views such as a "journal" style temporal view (which might also be useful for backup purposes). However, these extended metadata scenarios may beg a re-evaluation of the distributed versus centralized storage model.

In any case, the basic plan is squirel away the extra meta data, indexed by the string filename/directory-name in a hidden file. If the directory isn't writable by eve then no extended metadata will be written and the directory will default to the underlying uname/gname/mode -- however, if eve isn't able to access the files, the overlay won't be able to access the data -- so it's likely a moot point.

v9fs: Grave Robber's from Outerspace

Inferno Enabled

I've resurrected the Inferno Enabled sticker logo.

PNG Version:
SVG Versions are available in white and black.

v9fs: Grave Robber's from Outerspace

Infernal OLPC repository

I've reset up my Inferno git repository on http://git.9grid.us/git based on a mirror of Forsyth's SVN repository. Included now is my working OLPC modifications repository which contains both my core-Inferno patches as well as new mk rules for packaging. I'm going to move on to better OLPC applications support now, but there probably won't be an update on that until after next weekend.

v9fs: Grave Robber's from Outerspace

Infernal OLPC Packages Available

Finished working on the details on how to launch Inferno within the OLPC infrastructure and packaged an initial release which is now available via the OLPC wiki.

I need to take the sandbox I built it from and work out a merge into the mainline inferno-os repository. Basically this will amount to moving some bits into olpc specific directories and writing a distribution script which builds the bundle.

Next step after that will be to really go to work on customizing the environment for the OLPC, starting with a custom look and feel for the window manager and a few example applications. That will include supporting scaled true-type fonts and perhaps SVG support. I also need to work out some day-one scripts to help setup the right name space for the OLPC's restricted security model.

v9fs: Grave Robber's from Outerspace

OLPC - Hosted Inferno


Got my daughter an OLPC for Christmas, but have spent a lot of time playing with it myself. First order of biz was to load Inferno on it ;) I can run it from the shell quite happily, but am now trying to come up with a Sugar launcher and package for wider deployment. I may start with a simple sugar app that toggles between loading Inferno versus loading sugar. This should be much easier as it'll avoid interacting with the Python gorp. Another alternative would be to just hack the terminal application (which would then act as emu's console), but I'd like to try to find a way to tie it in better.

(UPDATE: launcher is working, now working on packaging)

v9fs: Grave Robber's from Outerspace

Inferno Running Byte Code on BG/L


Inferno is also running (hosted) on both Compute and I/O nodes.

v9fs: Grave Robber's from Outerspace

xsm support in

I've added the Xen shared memory channel bits to devshm and pushed the code to the inferno-power repository (although hypothetically this is a more generic feature). I really need to separate the devshm code from the power code, it just seemed more convenient to work on them in the same branch.

The Xen shared memory code only functions as a server right now. I just tested it with a slightly modified Libra client and everything seems cool. I need to revisit performance results to make sure I didn't screw anything up and work out the right way to modify the Libra launch scripts.

v9fs: Grave Robber's from Outerspace

devshm updates

Spent half the week chasing down a truly ridiculously stupid bug in the mmap code for devshm. I had intended to have the Xen based shared memory done by now and maybe even a Libra version of Inferno -- unfortunately that has all gone down the tubes for the time being.

The devshm code with SysV and mmap based shared memory transports is available off of the 9grid.us GIT repository, I need to write a man page before submitting a patch to the Inferno mailing list. Now I'm off to port the xen devchan interface.

v9fs: Grave Robber's from Outerspace

Linux-Power Inferno (take 2)

Nearly complete with a second take on my Linux-Power Inferno port. I've corrected (with Forsyth's help) many of the hacks I had rushed through the first time. I'm still a bit concerned with floating point and math libraries, but on the whole it seems functional and no longer uses pthreads so hopefully the SMP problems we were seeing will go away.

I'm trying for a clean merge against the Google Code inferno-os tip now, which should leave me plenty of time tomorrow to do a clean integration of the shared memory transport with shm, mmap, and xen back-ends.

v9fs: Grave Robber's from Outerspace

Collab

In lamenting not really knowing the various states of a distributed team working on Plan 9 related projects, I was thinking through expedient mechanisms for loosely keeping in touch. There are a bunch of tools that seem to help with distributed teams: phone calls, IM, IRC, VNC, Peer-to-peer and centralized file sharing, SCM, etc. I was thinking of architecting a simple suite of tools to provide basic fuctionality within a Plan 9 paradigm.

The cornerstone would be a modified version of faces, which checked against a centralized file system (or maybe just a centralized place with file systems bound from several different servers). People currently active ('logged into') within the collaboration framework would have their face appear. Mouse-over the face gives a quick bit on their current status (i.e. working on /sys/src/collab.c), right click pops up a menu of different options (chat, blog, view, share, mount, etc.) -- each of these would plumb an action to an appropriate tool (either in ACME or raw).
  • chat - simple point-to-point IM facility, I'm toying with the idea of walkie-talkie style audio chat as well.
  • blog - pops up the person's "blog" which really is just news(1), but personalized for that person, maybe work out something more complicated with wikifs later, but news(1) will do for now
  • view - If the person has shared a window with you, view will plumb it to a viewer -- I haven't quite decided how to handle different types of views, could just do VNC, but I'd rather have the ability to share windows -- or really what I want is the ability to share ACME windows and keep the sides in sync (like a distributed Zerox) -- ultimately, there's all sorts of granularity to sharing, but for now it'll be a single thing (kinda works like snarf) and it'll have to be integrated into ACME and other tools which use it.
  • share - really mostly described above -- this basically grants the person access to your current share
  • mount - pops up a window with the person's public namespace mounted -- this could just be their /usr directory, or it could be something more. Haven't quite decided how to handle this completely.
There will be a variety of command-line tools that allow you to post shares, export portions of your namespace into your public namespace, and allow you to easily update your blog or status.

I think the simple approach works for 1-to-1 communication, not sure how to handle 1-to-many or many-to-many without things getting complicated. There's also various ways you can build on this to add additional context (like maybe per-project collaboration spaces). They key thing is that everything works through file system mounts.

A big missing item is SCM, I have been toying with some ideas around a git-like mechanism with a venti back-end. For the most part this isn't an immediate requirement though, centralized file repository backed by venti is fine -- although it would be nice to have a formula or script for setting up replicas of such centrailized repositories.

v9fs: Grave Robber's from Outerspace

Plan9/Inferno Revision Control

I've been playing with the idea of some simple shell scripts that would provide git-like-service on-top of venti for use as wide-area source control. Essentially, to access a repository, you'd need access to the venti hosting the repository and the current score of the repo which could be published in a 9p partition or web page or whatever. Alternatively you could have the branches of the repo published (or pull the branches from the single score).

The scm-part of the repo would contain the branches, tags, and log messages. Branches and tags are just files with the particular score in them (not sure that there really is a difference between the two) -- might also squirell a description or something in the file.

Log messages would be in files named after the score of the tree that their commit represents. The log message would contain a parsable pointer to the parent score of the commit. Probably reasonable to organize the log messages in directories by date or something to keep things efficient. The logs themselves are probably best stored in venti, referenced by a "special" branch named log.

Then it's just a matter of writing some scripts to handle initializing a scm database, adding and removing files, commiting changes, and helper routines for doing diffs, merges, branching, etc. Also could have git-pull/git-push equivilent commands for syncing remote and local venti stores (if you wanted to have a local venti).

v9fs: Grave Robber's from Outerspace