[AFS3-std] Compression support for AFS vos dump - Specification

[Jeffrey Altman]

Peter Somogyi back almost three years ago proposed the following 
protocol enhancement for creating dump files containing compressed
data.  An implementation of this proposal is available for review 
in OpenAFS RT Ticket 17947.

Feedback is requested.  

Jeffrey Altman



Compression support for OpenAFS - specification

- additional param for "volserver" command line:
	-z [gzip:<level>|bz2:<level>]
	where level: [1..9|default]
	Example:
		/usr/afs/bin/volserver -z gzip:5

- new volserver must detect "not to be compressed" new flag in volume header:
	- VolumeDiskData.reserved1[0] & 1

- new "vos" setfields flag:
	- cmd line: compression {enabled,disabled}
	- "use" volintInfo.spare[2]&1 and volintInfo.spare[3]&1
	- modify VolSetInfo and ListOneVolume
	- modify "vos examine" to show this flag

- new volserver must provide 1 new RPC call in the interface:
	- RXAFS_GetCapabilities - same as on fs interface
	- 2 separate bits for zlib and bz2 support
	- new volserver must query this function (if exists) in VolForwardMultiple and VolForward
	- VolForwardMultiple: send compressed format only when ALL receivers support that

- new volserver must produce always compressed dump if the compression is turned on - at "vos dump"

- new dump format:
	volserver dumps:
		<0x01><DumpHeader>
		<0x02><VolumeHeader>
		<0x03><VNode1>
		[<0x06><length ::= 4><afs_uint32: flags]
		<0x05><unknown length::=0x80><CompressData>
		<0x04><DumpEndMagic>
		[EOF]

		where:
		- <CompressData> ::= <CompressHeader><CompressContent>
		- <CompressHeader> ::= <length><byte: CompressionType> //where length is sizeof(<CompressHeader>)-sizeof(<length>)
		- <CompressContent> ::= the zlib/bzip2 generated content
		- <VNode1> must be the first large vnode (error case if it doesn't exist):
			<VNode1> ::= <vnodeNumber><uniquifier>{<tag greater than 20><contents>}*
		- <length>: variable size length, 0x80=unknown (means that this section must be parsed to proceed)
			- If bit 7 is not set, bits 0-6 are the length
			- If bit 7 is set, bits 0-6 tell you how many bytes long the length is.
			The length follows immediately in MSB-first order.
			- The special value 0x80 means indefinite length.
			- 0xfe and 0xff would indicate a single-bit value, with the value stored in the low-order bit of the length.
		- after <0x06>, "extra flags" must be followed (tag with explicit length), containing one or more afs_uint32's
		(the first afs_uint32 will be transmitted into VolumeDiskData.reserved1[0] - for example the DONT_COMPRESS ::=0x01 flag)
		If there's no <0x06> tag, the volume's compression must be enabled.

	volserver parser:
		- unknown high-level tags treatment:
			- <0x06..0x60>: <length> awaited, and then <length> bytes must be skipped, write warning to VolserLog
			- <0x61..0x7a>: skip 4 bytes, write warning to VolserLog
			- the tag <0x7e> has no data by itself, but labels the next tag as "critical"
			- all of 0x7b-0x7f will be single-byte tags with no data (skip them if they're not "critical", otherwise error)
			- 0x80-0xff: error

		- <CompressData> mustn't appear more times in the dump. (It would cause more complex buffering implementation.)
		- no <CompressData> embedding supported

- include zlib and bzip2 (src/zlib and src/bzip2)
- "don't compress" flag must be transmitted only when the source volserver's compression is set, and each receiver volserver supports this flag (in case of "vos release"), or when "vos dump" was invoked

Peter Somogyi
2005-FEB-04

psomogyi@gamax.hu

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3355 bytes
Desc: S/MIME Cryptographic Signature
Url : http://michigan-openafs-lists.central.org/archives/afs3-standardization/attachments/20071029/9c30f211/smime.bin