Verbesserte Präsenz durch Screensharing für ein Werkzeug zur verteilten Paarprogrammierung (ThesisDPPXIII)
Improved presence through screen sharing for a distributed pair programming tool
(Bachelorarbeit)
von
Stephan Lau, betreut von Karl Beecher und
Stephan Salinger
Antrittsvortrag / Preparation presentation
is here
Ziele
Saubere und schnelle Implementierung von Screensharing in
Saros 
Planung / Approximate timeline
- registration: 17.11.2009
- delivery: 9.2.2010
- ill: 21.12.09 - 8.1.10 (19 days)
- Week 1-2 9. - 20.11
- base architecture and requirements for screensharing
- start to implement
- Week 3-4 23.11 - 4.12
- start integration in saros
- only passive features first
- Week 5-6 7.12 - 18.12
- Week 7-8 21.12 - 1.1.2010
- prepare feature for release
- alternative encoders
- test/optimize
- Week 9-10 4. - 15.1
- deal with scientific questions, theoretical preparation for thesis
- evaluate screensharing
- what has been done?
- how is screensharing accepted (by saros' users)?
- how (or does) screensharing improves saros?
- are there remaining problems?
- screensharing <→ collaboration awareness
- how are they related?
- can they be combined?
- …
- Week 11-12 18.1 - 5.2.
Anforderungen / Requirements
- functional
- client (consumer)
- see screen (or part) of remote desktop
- zoom into desktop
- useful when desktop has higher resolution than video
- click and/or point on the remote desktop
- send this activities to server
- resize video to size of player
- server/host (provider)
- provide local desktop
- capture desktop, put it in videostream
- execute actions send by client
- pretend overload of connection between server and client
- control bitrate of stream and adjust when necessary
- non-functional
- remote desktop is viewed in realtime or as fast as possible
- source of transmitted images and used encoder is replacable (by architecture)
- effective encoding
- fallback when native encoder not installed
Optional
Will be implemented when time is left.
- focus sharing on one or more windows
- support multi-monitor environments
- switch between screen and webcam
- record session, maybe with activities
- share some parts of eclipse only
- e.g. a console or something
- one host, many clients
- don't use slow codecs (e.g.
libx264) on slow machines
Anwendungsfälle / Use Cases
see
ThesisDPPXIIIUseCases
Implementierung
see
ThesisDPPXIIIProgramming
Literatur
see
ThesisDPPXIIILiteratur
Status
This layer adds the ability to define services/features which need pure streams (Java's
InputStream /
OutputStream).
Current state
running, but sending is not reliable enough
Working features
- session negotiation
- smooth termination of sessions
- configure characteristics of streams to meet non-functional requirements
- size of buffer
- minimal size of chunks transferred
- maximal delay between sending sending data
Problems
- =BinaryChannel= deadlocks sometimes → data lost, stream broken
- when an
OutputStream is closed which still contains data they will be lost
- reason:
MetaPacket send immediately, but data is queued
Concrete TODO's
- solve stream-close problem (s.a.)
- review errorhandling
- =StreamSessionOutputStream= : public methods to get buffer's state
- =SteramSessionInputSession= : count read bytes, public available
Nice-to-have features
- integrate
SubMonitor into streams methods
- not really necessary, but
DataTransferManager uses it
This is the main feature of this thesis.
It's purpose is to see the desktop of a participant in a saros-session.
The implementation is a general framework to supply any sequence of images
from one to another saros-user.
Current state
basically runs, a session can be started
Working features
- start a session, see remote desktop
- capture mouse- and keyboardevents in player
- switch between fullscreen and mouse following
- set up basic preferences via eclipses preferences
Problems
- xuggler
- resampling of images takes a lot of computation (more than compressing the picture) because it's not accelarated by hardware
- on mac only supported for 64-bit java 1.6
- good argument to implement different en-/decoder
- player
- draw and keep initial background
- scrollable or resize input
- desktop/screen
- zoom-in/-out when following mouse
- preferences
- validate new chosen values (e.g. invalid container for xuggler)
Concrete TODO's
- implement pure java de-/encoder
- monitor and throttle used bandwidth
- make GUI usable
- good icons for view and starting session
- fix mentioned problems in player
- preferences
- switch resolutions via combobox
- choose different en-/decoder
- stop and dispose session
- errorhandling
Nice-to-have features
Not sure I can make it for this thesis.
- execute click/type action on host (active screensharing)
- user-rights management in preferences
- record session (transmitted stream and activities)
Berichte
Vorbereitungsphase / Preparatory stage
Week ( 1.10 - 2.10.09)
Activities
- took part in testing of release 9.10.2
- search existing libraries/programms in java for screensharing or screencapture (on javaforge, berlios, freshmeat, sourgeforge)
Results
- invitationprocess seems to be buggy → not released yet
Problems
- not found many existing tools.. am I the first one using this in java!?!?
Goals
- get an overview to screensharing
- find some more javastuff concerning this topic
Week ( 5.10 - 9.10.09)
Activities
- still search some javaapis or solutions for screensharing
- dealt with videoencoding in java
- explored xuggler
- started prototype for screensharing based on xuggler
Results
- seems that I have build screenshare on my own
- use a simple videostream for transmitting the screen
- did not found other existing screensharingtools or apis for dynamic streaming of content
- chosen an api for videoencoding Xuggler
- based on ffmpeg
- wide range of supported codes
- active development
- fast encoding due to native code
- easy to install on windows, mac, linux with integrated and compiled ffmpeg
Problems
- videoencoding is very complex
- choose a preferred container and videoformat
- understand lots of possible settings an encoder uses
Goals
Week ( 12.10 - 16.10.09)
Activities
Results
- found 'hidden' settings for encoders in xuggler
Problems
- most settings applied not to codec h.264
- was ill from 13.10 to 16.10
Goals
Week ( 19.10 - 23.10.09)
Activities
- worked on prototype
- set up libx264 encoder to use cbr (constant bitrate encoding)
- meet the given bitrate and don't exceed it
- first checkin of prototype into svn
Results
- prototype
- client-server program, streaming via network
- dynamic resizing of inputframes to encoder
- console in server to change and view current enviroment
- change fps while streaming
- use vbv-buffer (in ffmpeg) to guarantee the bitratelimit (in some cases, esp. low bitrates, without vbv some peaks appeared)
Problems
- due to update of xuggler, 'libx264' needed other settings than default ones
- presets by ffmpeg did not work
- examples by xuggler used not cbr-mode
- bruteforce trying to find appropriate settings
- in cbr mode x264 has a noticable delay
- seems that encoder collects all frames between two keyframes, encodes them at once to keep given bitrate and finally pushes them into videostream
- xuggler updated to 3.3 on 21.10, so problems on setting up 'libx264' disappeared (see last week)
Goals
- release saros
- make weekly reports
- prototype
- deal with temporary low bandwidth
Week ( 26.10 - 30.10.09)
Activities
- improved prototype
- releasemanager in this cycle
- tested new release
Results
- prototype
- extended console
- threaded server
- change resolution while streaming → problem
- introduced and second stream for controlling the transmission
- client sends statistics about delay etc. to server
- releasemanager
- collected all changes since last release
- prepared branch for testing
Problems
- resizing the livestream is not possible
- used container (flv) does not support more than one videostream
- other containerformats don't seem to work
- either not streamable (mpeg4, …)
- or don't work w. xuggler? (mkv, ogg, …)
- setting a new framerate on the fly causes a big delay on client
- reason can't be figured out, maybe is the old timebase (based on old framerate) the reason
- try to drop frames instead of changing framerate
Goals
- prepare the presentation for thursday next week
- plan advance for the next weeks
- give an overview about screensharing
- details of planned implementation
- features
- requirements
- integration into saros
- find reason why some containerformats don't work in xuggler
- prototype
- deal with temporary low bandwidth
Working stage
Week ( 2.11 - 6.11.09)
Activities
Results
Problems
- not done
- find reason why some containerformats don't work in xuggler
Goals
- complete base architecture for screenshareapp
- send some bugreports to xuggler
Week( 9.11 - 13.11.09)
Activities
- worked on base architecture
- meeting with Stephan and Karl on wednesday
Results
- wrote usecases (ThesisDPPXIIIUseCases) and requirements
- developed modules (functionality, responsibilities)
- meeting
- shorten iterations to 2 weeks
- immediately integrate screensharing into saros
- some bugreports for xuggler
Problems
- this week was not really productive, i increase it!
Goals
- adapt plan due to meeting
- finish architecture
- start integration in saros
Week ( 16.11 - 20.11.09)
Activities
- preparation for implementation (requirements engineering)
- worked on implementation
- adapted plan
Results
- architecture and modules, see ThesisDPPXIIIProgramming
- started to code (see my repository)
- ported prototype to new architecture
- local testing via piped streams possible
Problems
- still some open questions for architecture
- who controls when a frame will be rendered, encoder or connectionmanager?
- what is needed at clients side to initiate and handle connections?
- need some details concerning saros, should be done this week
- how can i handle configurations?
Goals
- integrate into saros
- view for clients player
- bytestreams for connection between two users
Week( 23.11 - 27.11.09)
Activities
- programmed feature
- startup with saros
- get a network stream
Results
- programming
- hooked into picocotainer
- tried to get network stream
- entered tests 3.2 - 3.7 in testlink
Problems
- is not easy to get network stream
- find a way to get one via jingle, but this seems not to be accepted → meeting tuesday
Goals
- find solution for network architecture (meeting tuesday)
- let saros be packetbased, wrap a stream with them
- get a real stream via jingle/socks
- not completely my part, sandor and henning have to help me
- start ui
- player
- session initiation, negotiation
Week( 30.11 - 4.12.09)
Activities
- worked on streaming with saros
- assisted releasemanger
- tested saros for release
Results
- developed framework for streambased services
- simple filetransfer-test worked
Problems
Goals
- test streaming with video, finish it completely
- start ui stuff
Week( 7.12 - 11.12.09)
Activities
- worked at our new streaming interface
- played a bit with swt and eclipse views
- implemented file sending based on streaming
Results
- streaming: improved it
- performance (chunkSize and maximalDelay)
- reliability, send packets same order they arrive
- introduced soft stop
- UI
- created first view (canvas with simple statusbar)
- discovered swt: widgets, layouts, …
Problems
- not sure whether streaming is now reliable enough... spend already a lot of time there, but sometimes strange errors occur
- don't get so far to test videosharing with stream interface
Goals
- integrate view into videosharing
- create a StreamService for video, test it in saros
Week( 14.12 - 18.12.09)
Activities
- worked on view for playing video
- implemented example service for sending single files in a session
- improved streaming
- integration of videosharing-prototype into saros
Results
- videosharing
- created branch, moved classes
- created simple view for showing video with canvas and statusbar
- streaming
- can add serialized objects to metapackets
- developed improving concepts
- streams need to be closed
- output should block when input is not reading
- file sending
- show speed in task
- cancellable
Problems
- videoplayerview is swt, but images are awt: need to embed an awt canvas
- sometimes streaming hangs without obvious reason, need to figure out
Goals
- improve streaming (s.a.)
- esp. robustness (errorhandling, disposing)
- make videosharing work in saros
- embed awt-canvas in player
- create preferences (pages)
- catch input in player's canvas, send them to host
Week( 21.12 - 25.12.09)
Week( 28.12.09 - 1.1.10)
Week( 4.1 - 8.1.10)
Week( 11.1 - 15.1.10)
Activities
- finish screensharing and streaming for release
- documented state of work in wiki
- write paper about technical realization
Results
- screensharing
- disposing
- fast awt-canvas for displaying video
- catch input in videoplayer and send to host
- pure java encoder taken from smack-demo
- streaming
- errorhandling/disposing
- many fixes
- paper - 50% done
Problems
- alternative encoder is too slow, works only w. 1 FPS
- screensharing not ready for release
- streaming could be ready
- christopher fixes binarychannel
- must be reviewed (and afterwards fixed)
- any feature uses it
Goals
- make features ready for use for release next week
- optimize other encoder
- UI
- players toolbar
- resizing
- preference dialogues
- paper finished until wednesday
Week( 18.1 - 22.1.10)
Activities
- writing mentioned paper
- make screensharing ready for release
- make streaming ready for release
- reviewed some patches for release
Results
- paper done
- screensharing works
Problems
- seems that there's not enough to review both features
- maybe not released or only experimental?
- alternative encoder
- really bad performance, only 1 fps at low resolution, even at fast machines
- only core features in screensharing work
Goals
- release a new version for Saros
- finish streaming and sending file during session
Week( 25.1 - 29.1.10)
Activities
- helped releasing and testing saros 10.1.29
- worked on streaming and screensharing
Results
- streaming
- created patch for reviewing
- screensharing
- worked towards creating patch, not done yet
Problems
Goals
- next weeks
- get screensharing stable
- focus on thesis
- improve streaming according to reviews
- next week
- relax @ holiday in austria
Week( 1.2 - 5.2.10)
Week( 8.2 - 12.2.10)
Activities
- worked on thesis
- improved streaming
Results
- created outline
- read and chose some papers, see ThesisDPPXIIILiteratur
- streaming
- almost done, thread-pools are missing
Problems
Goals
- write parts about saros, pp, dpp and ca
- create new patch for streaming to be reviewed
Week (15.2 - 19.2.10)
Activities
- read some more papers
- prepared streaming for release
Results
- finished streaming patch, which will be reviewed for release
Problems
Goals
Week (22.2 - 26.2.10)
Activities
- adapted patch for streaming
Results
- streaming-patch
- adapted due to hennings and flo's review
Problems
- patch done too close to release, not released
Goals
- get finally better
- find some papers concerning video (esp. bandwidth-adaption)
- write parts about saros, pp, dpp, ca and related work
Kommentare
Wer will, wer will, wer hat noch nicht?
Ich würde mich über jede Anregung, Anmerkung, Kritik freuen.
