Thursday, June 19, 2008
9p/virtio Slides Available
The slides I presented at KVM forum on using 9p over virtio to provide a paravirtualized file service are available on the KVM wiki
other page actions:
Friday, April 25, 2008
xcpu quick start
xcpu quick start for impatient people like me
(aka xcpu quickstart) (More information about xcpu is available from http://www.xcpu.org) Make Sure You Have The Pre-Reqs (on Ubuntu 8.04) sudo apt-get install libelfg0-dev sudo apt-get install libssl-dev Acquire the Source
(aka xcpu quickstart) (More information about xcpu is available from http://www.xcpu.org) Make Sure You Have The Pre-Reqs (on Ubuntu 8.04) sudo apt-get install libelfg0-dev sudo apt-get install libssl-dev Acquire the Source
svn co svn://xcpu.org/sxcpu/trunk sxcpu
cd sxcpu
make
sudo make install
To try things out on the same host (single system, no client server) % su % ssh-keygen -t rsa -f admin_key % mkdir /etc/xcpu % cp admin_key* /etc/xcpu % xcpufs & % xgroupset 127.0.0.1 root 0 % xuserset 127.0.0.1 root 0 root /root/.ssh/id_rsa.pub % xrx 127.0.0.1 /bin/date To setup a pre-existing cluster with xcpu On the front end machine (as root): % ssh-keygen -t rsa -f admin_key % mkdir /etc/xcpu % cp admin_key* /etc/xcpu (copy admin_key to each back end node - below example with ssh) % ssh root@each_back_end_server mkdir /etc/xcpu % scp admin_key root@each_back_end_server:/etc/xcpu To setup just the compute node (back-end server) % xcpufs & Then back on front-end node: % sudo xgroupset % sudo xuserset ~/.ssh/id_rsa.pub
% xrx /bin/date
Wednesday, April 16, 2008
trans_fd_simple
Did some really basic performance checks on trans_fd_simple compared to the older code and comparing single lock with reader/writer lock protection on the socket. Turns out my thinking with reader/writer lock was flawed and there are race cases (hence only the single threaded result). Long and short of it appears to be sort of what we figured, that the simple approach does better on single threaded workload due to less overhead, but does much worse on multi-threaded due to added latency and what not. I ran against npfs as well just to check if earlier thinking about whether or not multi-threaded npfs added much -- it appears to add nothing but overhead in this case.
This particular case was from a qemu hosted linux talking to the simulation host server -- which isn't particularly good. I'm going to replicate this test on the blade regression cluster, but wanted to do a quick check before moving on to scatter/gather implementations.
This particular case was from a qemu hosted linux talking to the simulation host server -- which isn't particularly good. I'm going to replicate this test on the blade regression cluster, but wanted to do a quick check before moving on to scatter/gather implementations.
Wednesday, April 16, 2008
Service Oriented File Systems
Its fun to abstract and think about web2.0 concepts in terms of Plan 9 concepts. Abstract away the gorp that makes up their implementations and think about some of the fundamental concepts. When interfaces are files, mashups are binds.
The larger picture here is about keeping this simple, language independent, and client independent. I'd like an infrastructure that I could build simple web services out of that can be viewed with a browser, or with a rich-client (like acme plug-ins). I'd like the front-end of these services to be defined in the same way the Octopus approach defines normal GUI's (except from a web-browser, the widgets are implemented by Java script). I'd like the back-end of these to be akin to big table and the chubby lock service, with a back-back-end that is simply Venti versus a database or XML.
I'll admit I'm probably naive about all of this, not having really worked on client-facing applications for some time, but I think it would be a fun thought exercise to go through and perhaps a nice refresher to actually try and implement. While part of the goal is to keep things language independent, I'll probably do things under Inferno to keep them portable for me and open up the potential for interaction with the Octopus crowd.
Of course, this all falls under the copious spare time which I have a definite deficit of -- however, I think dedicating one day a week is doable and I should be able to combine this with my constant desire to get back and look at collaborative facilities (aka warren) under Plan 9 and Inferno.
The larger picture here is about keeping this simple, language independent, and client independent. I'd like an infrastructure that I could build simple web services out of that can be viewed with a browser, or with a rich-client (like acme plug-ins). I'd like the front-end of these services to be defined in the same way the Octopus approach defines normal GUI's (except from a web-browser, the widgets are implemented by Java script). I'd like the back-end of these to be akin to big table and the chubby lock service, with a back-back-end that is simply Venti versus a database or XML.
I'll admit I'm probably naive about all of this, not having really worked on client-facing applications for some time, but I think it would be a fun thought exercise to go through and perhaps a nice refresher to actually try and implement. While part of the goal is to keep things language independent, I'll probably do things under Inferno to keep them portable for me and open up the potential for interaction with the Octopus crowd.
Of course, this all falls under the copious spare time which I have a definite deficit of -- however, I think dedicating one day a week is doable and I should be able to combine this with my constant desire to get back and look at collaborative facilities (aka warren) under Plan 9 and Inferno.
Monday, April 14, 2008
v9fs updates
With the 2.6.25 merge window rapidly approaching, it looks like I'm going to have to hold off on merging the reorganization until 2.6.26. At the moment, I'm still fiddling with the transport API and specifically the rewrite of the file descriptor based transport which is used for sockets, named pipes, etc. With a 2.6.26 target, I should be able to get the async and scatter-gather rewrites complete as well as merge the in-kernel server code from lucho.
As part of the exercise I've created a "simple" version of the code which is synchronous and single threaded (to match spfs). My motivation was to scale back and figure out what the core pieces were in the fd transport, as well as to have a useful performance baseline to measure the overhead (and advantages) of a more asynchronous model. A nice side-effect is that I also have a really simple and straightforward transport module which can be used as an example for those wanted to write their own transports. I'm not quite certain whether I want to merge the simple transport mainline though -- although I may merge it into trans_fd in order to maintain it as a baseline case -- it really amounts to a different request member function which could easily be selected via an option.
In parallel I've setup a few blades in order to measure performance, and perhaps more importantly performance regression. When complete, I hope to publish historical performance regression results back to kernel 2.6.14. At the moment I'm trying to do all collection of results and graphs inside Google docs, which should make an easy place to keep things up to date and simultaneously publish the data. I need to look around to see if anyone has hooks for the Google Data API so I can have my regression framework update the results automatically instead of me having to go in and manually enter the data.
As part of the exercise I've created a "simple" version of the code which is synchronous and single threaded (to match spfs). My motivation was to scale back and figure out what the core pieces were in the fd transport, as well as to have a useful performance baseline to measure the overhead (and advantages) of a more asynchronous model. A nice side-effect is that I also have a really simple and straightforward transport module which can be used as an example for those wanted to write their own transports. I'm not quite certain whether I want to merge the simple transport mainline though -- although I may merge it into trans_fd in order to maintain it as a baseline case -- it really amounts to a different request member function which could easily be selected via an option.
In parallel I've setup a few blades in order to measure performance, and perhaps more importantly performance regression. When complete, I hope to publish historical performance regression results back to kernel 2.6.14. At the moment I'm trying to do all collection of results and graphs inside Google docs, which should make an easy place to keep things up to date and simultaneously publish the data. I need to look around to see if anyone has hooks for the Google Data API so I can have my regression framework update the results automatically instead of me having to go in and manually enter the data.
Tuesday, February 26, 2008
GSoC 2008 Ideas
My Plan 9/Inferno summer of code ideas (otherwise known as projects I wish I had more time for) - projects should take no longer than 5 weeks to complete (including ramp up, debug, and packaging) -- conservatively projects should be completable in 3 weeks by experienced P9/Inferno developers and 2 weeks by folks unfamiliar with p9/inferno development environments.
- mount.9P helper program for v9fs including packaging for debian and fedora -
- hueristic for determining transport type
- DNS support for resolving hostname <->ip address
- ssh-based tunneling support (will require server work as well)
- man pages
- debian packaging
- rpm packaging
- (stretch) authentication support (w/p9p)
- (stretch) authentication support (w/o p9p)
- (stretch) integration with idmap solution
- since this is relatively straightforward it will really need to be "super" version including support for ssh tunneling, authentication, etc.
- idmap solution for v9fs - maps local uids/gids/error-codes to strings
- userspace daemon to provide mapping
- hooks in v9fs to use mapping
- (stretch) synthetic file server approach to update mappings
- OLPC Inferno Environment
- fontfs - on-demand ttf solution
- metafs - abstract metadata from underlying file system
- (stretch) Integrate OLPC translation solution for Inferno
- (stretch) OLPC oriented GUI toolkit, window manager, and toolbar
- (stretch) Inferno approach to collaboration using file systems
- Source Control Management based on Venti backend
- single branch version tracking with associated log file
- add new files/directories
- commit changes
- checkout specific version #
- (stretch) support multiple branches
- (stretch) support repository sync (push/pull)
- (stretch) support three-way merge
- wrapper for p9p vbackup to make it more user friendly
- wrapper which tracks venti scores based on volume being backed up
- GUI admin tool
- which sets up venti in partition(s) or with files
- which assists in configuring backup intervals and volumes
- (stretch) time-traveler like GUI for navigating/searching backups
- (stretch) support for pruning and/or merging snapshots
- GSoCFS for managing future community involvement with GSoC
- synethtic file system and web interface
- posting project ideas
- voting for projects
- registering student interest in projects
- project milestone tracking
- project blogs and wikis
- post-summer project success metrics (subjective and objective)
- (stretch) syndication points for community monitoring
- (stretch) potential integration with SCM
- (stretch) potential integration with some form of chat
- (stretch) potential integration with name space sharing
Thursday, January 31, 2008
inferno-olpc circa 1/28/08
This week's update to inferno-olpc includes OLPC-optimized fonts and a customized logon with two extensions:
I had a larger agenda that included social-network style "profiles" and integrated security/password setup. But there were too many corner cases to test and "bullet-proof" so I decided to hold that off for later and maybe just incorporate those functions into separate applications.
I'm still a little uncertain of how I want to handle authentication and security. On one side, there could be a central "schoolserver" to setup authentication against. However, as most of the current users don't have such a server, having a pure peer-to-peer mode is probably preferable. Given that every laptop will be an auth-server then, do we setup a laptop wide authentication setup, or do we do a per user authentication setup. The later would seem to make more sense to me as the user-provided password wouldn't have to be stored anywhere and could be use to setup keyfs and the user's own authentication secrets. Other users "registering" with that user for the purpose of collaboration could then provide their own secrets in his keydb, etc. I'm not sure if that makes sense or if there is a better model I should be considering.
The file system security is another pain. Inferno users won't necessarily equal underlying system users (a common problem). This will be made worse with the new OLPC security enforcement which may have different uid's for each "instance" of Inferno (but the same "gid"). I think the right thing to do is a new version of devfs which always keeps group permissions open in the underlying fs and uses extended attributes or some other mechanism to maintain Inferno uid's and gid's for the files.
- it binds the Inferno activity "writable" area to important areas of the Inferno name space which the user may want to write to (/usr, /services, /keydb, /tmp). This will be necessary for the next OLPC update which will start enforcing security (including only allowing writing to the activity writable area of the file system. Also bundled is a day-one scenario which will setup template directories to bind from and copying relevant files.
- when logon can't find a user it will now prompt to create the user by copying the template /usr/inferno directory to /usr/username. This can probably be generalized back into the standard Inferno base.
I had a larger agenda that included social-network style "profiles" and integrated security/password setup. But there were too many corner cases to test and "bullet-proof" so I decided to hold that off for later and maybe just incorporate those functions into separate applications.
I'm still a little uncertain of how I want to handle authentication and security. On one side, there could be a central "schoolserver" to setup authentication against. However, as most of the current users don't have such a server, having a pure peer-to-peer mode is probably preferable. Given that every laptop will be an auth-server then, do we setup a laptop wide authentication setup, or do we do a per user authentication setup. The later would seem to make more sense to me as the user-provided password wouldn't have to be stored anywhere and could be use to setup keyfs and the user's own authentication secrets. Other users "registering" with that user for the purpose of collaboration could then provide their own secrets in his keydb, etc. I'm not sure if that makes sense or if there is a better model I should be considering.
The file system security is another pain. Inferno users won't necessarily equal underlying system users (a common problem). This will be made worse with the new OLPC security enforcement which may have different uid's for each "instance" of Inferno (but the same "gid"). I think the right thing to do is a new version of devfs which always keeps group permissions open in the underlying fs and uses extended attributes or some other mechanism to maintain Inferno uid's and gid's for the files.
Thursday, January 31, 2008
freefontfs - a styxserver inferno-lab
I've decided doing the dynamic free font file system would be a more gentle introduction to using the new Inferno styxservers modules than a full-blown filterfs like what we plan for metafs.
The most appropriate example would appear to be mntgen.b (which allows for dynamic walk based creation in /n). It uses both the styxservers facilities and the navigator and has a dynamic nature similar to what I want for freefontfs.
Fonts are comprised of two types of file, the base file (..font) and subfonts (..). We only want to generate (and cache) files as they are accessed. The base available 's will be derrived from crawling a directory for ttf files or perhaps from a configuration file.
Following mntgen, it looks like it would be much more simple to keep it as a single level hierarchy (much like /n) -- this may be possible as all the necessary information is encoded in the filename. We just need to parse the single walk correctly.
One question is whether we should prepopulate the cache with a base font size for every ttf available. This seems like a reasonable idea, but probably wouldn't even be necessary in many situations -- perhaps we can leave it as a config option.
The most appropriate example would appear to be mntgen.b (which allows for dynamic walk based creation in /n). It uses both the styxservers facilities and the navigator and has a dynamic nature similar to what I want for freefontfs.
Fonts are comprised of two types of file, the base file (..font) and subfonts (..). We only want to generate (and cache) files as they are accessed. The base available 's will be derrived from crawling a directory for ttf files or perhaps from a configuration file.
Following mntgen, it looks like it would be much more simple to keep it as a single level hierarchy (much like /n) -- this may be possible as all the necessary information is encoded in the filename. We just need to parse the single walk correctly.
One question is whether we should prepopulate the cache with a base font size for every ttf available. This seems like a reasonable idea, but probably wouldn't even be necessary in many situations -- perhaps we can leave it as a config option.
Thursday, January 31, 2008
metafs
Since I wrote about it in my other post, I might as well explain metafs. Basically in order to accommodate the OLPC file system security model I need to maintain uid, gid, and mode bits for Inferno's view of the underlying file system separately. This is actually a common need within hosted Inferno when you don't necessarily want to keep Inferno user's sync'd with system users. In any case, this amounts to an overlay file system, which stores away the additional metadata.
While I am tempted to keep a central database of the metadata, the most pragmatic solution would seem to either use extended attributes or a hidden file in each directory. Since extended attributes are somewhat distasteful and not available anywhere, I'm going to start with the per-directory hidden file.
In the future this same mechanism might be extendable to implement semantic file system extensions (or extended attributes for that matter) -- allowing for tagged organization and searching of the file system as well as alternative views such as a "journal" style temporal view (which might also be useful for backup purposes). However, these extended metadata scenarios may beg a re-evaluation of the distributed versus centralized storage model.
In any case, the basic plan is squirel away the extra meta data, indexed by the string filename/directory-name in a hidden file. If the directory isn't writable by eve then no extended metadata will be written and the directory will default to the underlying uname/gname/mode -- however, if eve isn't able to access the files, the overlay won't be able to access the data -- so it's likely a moot point.
While I am tempted to keep a central database of the metadata, the most pragmatic solution would seem to either use extended attributes or a hidden file in each directory. Since extended attributes are somewhat distasteful and not available anywhere, I'm going to start with the per-directory hidden file.
In the future this same mechanism might be extendable to implement semantic file system extensions (or extended attributes for that matter) -- allowing for tagged organization and searching of the file system as well as alternative views such as a "journal" style temporal view (which might also be useful for backup purposes). However, these extended metadata scenarios may beg a re-evaluation of the distributed versus centralized storage model.
In any case, the basic plan is squirel away the extra meta data, indexed by the string filename/directory-name in a hidden file. If the directory isn't writable by eve then no extended metadata will be written and the directory will default to the underlying uname/gname/mode -- however, if eve isn't able to access the files, the overlay won't be able to access the data -- so it's likely a moot point.
Wednesday, January 23, 2008
Inferno Enabled
Monday, January 21, 2008
Infernal OLPC repository
I've reset up my Inferno git repository on http://git.9grid.us/git based on a mirror of Forsyth's SVN repository. Included now is my working OLPC modifications repository which contains both my core-Inferno patches as well as new mk rules for packaging. I'm going to move on to better OLPC applications support now, but there probably won't be an update on that until after next weekend.
Monday, January 14, 2008
Infernal OLPC Packages Available
Finished working on the details on how to launch Inferno within the OLPC infrastructure and packaged an initial release which is now available via the OLPC wiki.
I need to take the sandbox I built it from and work out a merge into the mainline inferno-os repository. Basically this will amount to moving some bits into olpc specific directories and writing a distribution script which builds the bundle.
Next step after that will be to really go to work on customizing the environment for the OLPC, starting with a custom look and feel for the window manager and a few example applications. That will include supporting scaled true-type fonts and perhaps SVG support. I also need to work out some day-one scripts to help setup the right name space for the OLPC's restricted security model.
I need to take the sandbox I built it from and work out a merge into the mainline inferno-os repository. Basically this will amount to moving some bits into olpc specific directories and writing a distribution script which builds the bundle.
Next step after that will be to really go to work on customizing the environment for the OLPC, starting with a custom look and feel for the window manager and a few example applications. That will include supporting scaled true-type fonts and perhaps SVG support. I also need to work out some day-one scripts to help setup the right name space for the OLPC's restricted security model.
Monday, December 31, 2007
OLPC - Hosted Inferno
Got my daughter an OLPC for Christmas, but have spent a lot of time playing with it myself. First order of biz was to load Inferno on it ;) I can run it from the shell quite happily, but am now trying to come up with a Sugar launcher and package for wider deployment. I may start with a simple sugar app that toggles between loading Inferno versus loading sugar. This should be much easier as it'll avoid interacting with the Python gorp. Another alternative would be to just hack the terminal application (which would then act as emu's console), but I'd like to try to find a way to tie it in better.
(UPDATE: launcher is working, now working on packaging)
Friday, November 30, 2007
Service Oriented Synthetic File Systems
I was doing some thinking about how larger synthetic file systems are structured and implemented under Plan 9 (or using 9P in general).
My toy-use-case for thinking about it was developing a synthetic-file-server based service similar to SourceForge -- where you have many different projects, each with different component services (bug-tracking, version-control, blogs, wiki, forums, etc.)
You could do it by running lots of different servers (per project) and binding them into a name space and then exporting, but that seems kinda awful (particularly if you have dozens of projects you are using the same tool to track). Much better to have a single server running which is able to accommodate multiple projects, each with multiple file services underneath.
Obsession with qid-spaces ends up being distracting, with the way fids work, we shouldn't need to segment the space so rigidly, it seems to me that qids only really need to be unique per directory, outside of that you can track which "module" of the hierarchy you are in by saving some state in the server-side fid tracking structure as you traverse the hierarchy.
dot-dot makes things a little more complicated, but the solution used inside the kernel can be used within file servers as well (namely, keep the whole path in the service-side fid tracking structure).
What's really nice is you can develop plug in file server modules which allow you to compose much more complex single-server-services out of many different file system components -- without worrying about how the qid space is split up and managed.
My toy-use-case for thinking about it was developing a synthetic-file-server based service similar to SourceForge -- where you have many different projects, each with different component services (bug-tracking, version-control, blogs, wiki, forums, etc.)
You could do it by running lots of different servers (per project) and binding them into a name space and then exporting, but that seems kinda awful (particularly if you have dozens of projects you are using the same tool to track). Much better to have a single server running which is able to accommodate multiple projects, each with multiple file services underneath.
Obsession with qid-spaces ends up being distracting, with the way fids work, we shouldn't need to segment the space so rigidly, it seems to me that qids only really need to be unique per directory, outside of that you can track which "module" of the hierarchy you are in by saving some state in the server-side fid tracking structure as you traverse the hierarchy.
dot-dot makes things a little more complicated, but the solution used inside the kernel can be used within file servers as well (namely, keep the whole path in the service-side fid tracking structure).
What's really nice is you can develop plug in file server modules which allow you to compose much more complex single-server-services out of many different file system components -- without worrying about how the qid space is split up and managed.
Wednesday, October 03, 2007
Applying mount namespaces (in Linux)
Decent developer works article describing how to use private name spaces in Linux including the new shared subtree stuff and examples about how to create per-login or per-user (where's per-process?) name spaces.
"Applying mount namespaces" IBM Developerworks
Per-process would be fairly easy to do given the techniques described in the paper -- we just need to work through an example, although it may be easier to do inside the source code of a particular shell.
"Applying mount namespaces" IBM Developerworks
Per-process would be fairly easy to do given the techniques described in the paper -- we just need to work through an example, although it may be easier to do inside the source code of a particular shell.
Saturday, August 25, 2007
v9fs performance versus NFS
I've been doing experiments looking at using 9p between partitions for file system access - focusing on optimizing a shared memory transport and eventually a cooperative cache.
However, I ran some baselines between partitions using the virtual network and it seems our performance has slid a bit.
Here's the results from 2 years ago using a gigabit network between two hosts:
And here is the result between two partitions on KVM comparing NFS, 9p, and local virtualized disk: (9p cached is loose cache mode, but since the working set is double the memory of the partition this actually worked against it):
Put block and get block are equivilent to the write_block and read_block of the older graph. Where 9p used to edge out NFS on reads, we now are a good deal worse. Our write performance looks like its as bad as it used to be and rewrite is worse.
Overall, we've had a big slide -- there has been two rewrites of the transport and support for multi-threaded workloads (which this workload doesn't exercise). This is also against a different server than original.
I need to go over my methodology a bit, but I think its probably worth revisiting performance issues.
However, I ran some baselines between partitions using the virtual network and it seems our performance has slid a bit.
Here's the results from 2 years ago using a gigabit network between two hosts:
And here is the result between two partitions on KVM comparing NFS, 9p, and local virtualized disk: (9p cached is loose cache mode, but since the working set is double the memory of the partition this actually worked against it):
Put block and get block are equivilent to the write_block and read_block of the older graph. Where 9p used to edge out NFS on reads, we now are a good deal worse. Our write performance looks like its as bad as it used to be and rewrite is worse.Overall, we've had a big slide -- there has been two rewrites of the transport and support for multi-threaded workloads (which this workload doesn't exercise). This is also against a different server than original.
I need to go over my methodology a bit, but I think its probably worth revisiting performance issues.
Friday, June 22, 2007
Blue Gene Project Pages
I've setup an more official website to put publications and presentations relating to the DOE sponsored Plan 9 on Blue Gene work. You can get to it here and I'll be updating it over the next few days.
Monday, June 18, 2007
Inferno Running Byte Code on BG/L
Inferno is also running (hosted) on both Compute and I/O nodes.
Monday, June 18, 2007
Torus and tree networks working on Plan 9 BG/L port
We finally worked out the remaining infrastructure issues and are now able to cpu(1) to both I/O nodes (over the Ethernet) and cpu nodes (over the tree network). The cpu nodes are also able to talk to each other over the torus network.We will be attempting a live demo during the USENIX poster session, so drop by and play with Plan 9 running on a Blue Gene.
Friday, May 11, 2007
9p and Linux Articles
By request, I've posted the two "semi-complete" drafts of a 5 paper series that intended to describe using and programing with 9p under Linux using v9fs and spfs. These are very fragmented drafts and somewhat out of date. If anyone is interested in finishing them off, please contact me. The original idea was to publish them on IBM Developerworks.
Page 1 | Next >>