On Thu, Oct 14, 2010 at 10:30:39AM -0700, Ryan Hobbs wrote:

> I may be all the way in left field, but I wanted to get your opinion 
> on a few items pertaining to sockets and the send/recv buffers.  To 
> your knowledge for very large files is there any advantage to 
> adjusting the socket send and recv buffers on both ends to enhance 
> performance?

i assume the question is really:

  'how to i used tcp to move images from the west coast to the east
     coast as quickly as possible' ?

	 you can increase socket buffers to get a large BDP (bandwidth-delay
	 product) but it probably won't work as wel as you want/need, expecially if you drop things

	 increasing socket buffers using setsockopt isn't all you need to do, and kernel tweaking of and some people suggest probably won't help that much either (recent kernels are pretty good already)



	 ... testing (right now) against a machine in Alanta, GA ...



	 with no tweaks at all when i used tcpdump to watch a connection i see a scale of 7 (meaning 128 times, the wscale is a power of 2) and a window of about 5792

	 the BDP is going to be (128*5792 bytes) / (69 ms) about 10MB/s

	 however, the window can ramp up higher, it starts low and increases over time; a test here sucking down 48MB of data shows it start there and ramp up to 24000 towards the end, give me a BDP of about 44MB/s

	 that's seem about right as the kernel as a 4MB (by default) limit on these buffers (google tcp_wmem and tcp_rmem and realize 95% of people claim is wrong); increasing that is or marginal value unless you really are close to that limit


	 so, tweaking things a little (increasing wmem & rmem as needed) i can get maybe 40% more performance before my link speed here limits me, at that point my BDP wasn't much higher though, 16231*256 (wscale 8 as a result of tcp tweaks) giving me an upper bound of 60MB/s


	 if you drop a packet, things back off quite quickly and have to ramp up again, i see thate here as my testing is to my laptop and network here is a bit naff


		 ok, this email took a bit long as it poorly structured right now, i was basically trying to show that tweaking tcp socket options and even kernel parameters won't get you as far as you want, you need to do other things


		 i had a whoe lot of other notes here on what you probably should be looking at, but i kinda want to sit of them for now until i get a chance to publicly talk about htem and/or give you code (i don't trust paul at all sorry)

		 > Secondly with regards to partinage-ng have you ever used the option 
		 > partimage-ng -blocksize?  Overall I would like to squeeze as much as I 
		 > can and I am wondering with your vast background here if you may have 
		 > some input on this.

		 i played with it a little, under 32K it didn't help image size much, this is somehwat expected, data tends to clump/cluster as that's how filesystems usually work

		 much larger cost some space and gained a little performance, teaking this is probably worth mmaybe 10% at most, where as there are factors of 5 to 10x elsewhere you can gain



		 assuming this moving GSIs about, i have a serious request

		 can i get the raw GSIs that you care most about that are public
		 somehow?  in whatever format?   i want to do analysis of the blocks in
		 order to determine end-end uniqueness (this should be a pretty big clue in itself)