Log inRegister

Impressum

You are here: SE » ThesesHome » ThesisDPPXIII

Verbesserte Präsenz durch Screensharing für ein Werkzeug zur verteilten Paarprogrammierung (ThesisDPPXIII)

09 Aug 2023 - 17:03 | Version 18 | UnknownUser

Improved presence through screen sharing for a distributed pair programming tool (Bachelorarbeit)

von Stephan Lau, betreut von Karl Beecher und Stephan Salinger

Antrittsvortrag / Preparation presentation

is here

Ziele

Saubere und schnelle Implementierung von Screensharing in Saros

Planung / Approximate timeline

registration: 17.11.2009
delivery: 9.2.2010
ill: 21.12.09 - 8.1.10 (19 days)

Week 1-2 9. - 20.11
- base architecture and requirements for screensharing
- start to implement
Week 3-4 23.11 - 4.12
- start integration in saros
  - UI
  - bytestream via XMPP
- only passive features first
Week 5-6 7.12 - 18.12
- additional features
  - active screensharing
Week 7-8 21.12 - 1.1.2010
- prepare feature for release
- alternative encoders
- test/optimize
Week 9-10 4. - 15.1
- deal with scientific questions, theoretical preparation for thesis
  - evaluate screensharing
    - what has been done?
    - how is screensharing accepted (by saros' users)?
    - how (or does) screensharing improves saros?
    - are there remaining problems?
  - screensharing <→ collaboration awareness
    - how are they related?
    - can they be combined?
    - …
Week 11-12 18.1 - 5.2.
- - final work on thesis

Anforderungen / Requirements

functional
- client (consumer)
  1. see screen (or part) of remote desktop
  2. zoom into desktop
    - useful when desktop has higher resolution than video
  3. click and/or point on the remote desktop
    - send this activities to server
  4. resize video to size of player
- server/host (provider)
  1. provide local desktop
    - capture desktop, put it in videostream
  2. execute actions send by client
  3. pretend overload of connection between server and client
    - control bitrate of stream and adjust when necessary
non-functional
1. remote desktop is viewed in realtime or as fast as possible
2. source of transmitted images and used encoder is replacable (by architecture)
3. effective encoding
  - use xuggler
4. fallback when native encoder not installed

Optional

Will be implemented when time is left.

focus sharing on one or more windows
support multi-monitor environments
switch between screen and webcam
record session, maybe with activities
share some parts of eclipse only
- e.g. a console or something
one host, many clients
don't use slow codecs (e.g. libx264) on slow machines
- ability to detect this

Anwendungsfälle / Use Cases

see ThesisDPPXIIIUseCases

Implementierung

see ThesisDPPXIIIProgramming

Literatur

see ThesisDPPXIIILiteratur

Status

StreamServices

This layer adds the ability to define services/features which need pure streams (Java's InputStream / OutputStream).

Current state

running, but sending is not reliable enough

Working features

session negotiation
smooth termination of sessions
configure characteristics of streams to meet non-functional requirements
- size of buffer
- minimal size of chunks transferred
- maximal delay between sending sending data

Problems

=BinaryChannel= deadlocks sometimes → data lost, stream broken
when an OutputStream is closed which still contains data they will be lost
- reason: MetaPacket send immediately, but data is queued

Concrete TODO's

solve stream-close problem (s.a.)
review errorhandling
=StreamSessionOutputStream= : public methods to get buffer's state
=SteramSessionInputSession= : count read bytes, public available

Nice-to-have features

integrate SubMonitor into streams methods
- not really necessary, but DataTransferManager uses it

ScreenSharing (or VideoSharing)

This is the main feature of this thesis. It's purpose is to see the desktop of a participant in a saros-session. The implementation is a general framework to supply any sequence of images from one to another saros-user.

Current state

basically runs, a session can be started

Working features

start a session, see remote desktop
capture mouse- and keyboardevents in player
- switch between fullscreen and mouse following
set up basic preferences via eclipses preferences

Problems

xuggler
- resampling of images takes a lot of computation (more than compressing the picture) because it's not accelarated by hardware
  - try to resample on our own, not by xuggler (good hint http://forums.sun.com/thread.jspa?threadID=5272305)
- on mac only supported for 64-bit java 1.6
  - good argument to implement different en-/decoder
player
- draw and keep initial background
- scrollable or resize input
desktop/screen
- zoom-in/-out when following mouse
preferences
- validate new chosen values (e.g. invalid container for xuggler)

Concrete TODO's

implement pure java de-/encoder
- promising possibilities
  - screenshare demo in Smack org.jivesoftware.smackx.jingle.mediaimpl.sshare used this as statrting point
  - http://dayonhome.sourceforge.net Dayon!
    - hard to understand, poorly commented
    - seems that pictures are only gray
  - http://java-remote-control.googlecode.com Java Remote Control
    - complex structure, is based on http://code.google.com/p/onestonessoup-xapp/ XAPP
    - poorly commented
monitor and throttle used bandwidth
make GUI usable
- good icons for view and starting session
- fix mentioned problems in player
preferences
- switch resolutions via combobox
- choose different en-/decoder
stop and dispose session
errorhandling

Nice-to-have features

Not sure I can make it for this thesis.

execute click/type action on host (active screensharing)
- user-rights management in preferences
record session (transmitted stream and activities)

Berichte

Vorbereitungsphase / Preparatory stage

More ...

Week ( 1.10 - 2.10.09)

More ...

Activities

took part in testing of release 9.10.2
search existing libraries/programms in java for screensharing or screencapture (on javaforge, berlios, freshmeat, sourgeforge)
- found
  - http://krut.sourceforge.net/
    - screengrabber, record desktopsession to mov-files
  - http://jshot.info/
    - take simple screenshots

Results

invitationprocess seems to be buggy → not released yet

Problems

not found many existing tools.. am I the first one using this in java!?!?

Comments

Goals

get an overview to screensharing
find some more javastuff concerning this topic

Week ( 5.10 - 9.10.09)

More ...

Activities

still search some javaapis or solutions for screensharing
dealt with videoencoding in java
- found
  - http://java.sun.com/javase/technologies/desktop/media/jmf/2.1.1/download.html
    - standard in java videoencoding
    - uses old, uneffective codes
    - since 2004 not developed further
  - http://de.sourceforge.jp/projects/sfnet_h264avcjavaenco/
    - pure h264 encoding in java… should be slow
  - http://www.alphaworks.ibm.com/tech/tk4mpeg4
    - mpeg4 aka h.263 coder in java
  - http://netmind.hu/ogg
    - just containerencoder without codecs
  - http://jffmpeg.sourceforge.net
    - ffmpegwrapper for java
    - extends jmf
  - http://www.xuggle.com/xuggler/
    - ffmpegwrapper for java
  - http://fmj-sf.net/index.php
    - ffmpegwrapper for java
explored xuggler
started prototype for screensharing based on xuggler

Results

seems that I have build screenshare on my own
- use a simple videostream for transmitting the screen
did not found other existing screensharingtools or apis for dynamic streaming of content
chosen an api for videoencoding Xuggler
- based on ffmpeg
  - wide range of supported codes
  - active development
  - fast encoding due to native code
- easy to install on windows, mac, linux with integrated and compiled ffmpeg

Problems

videoencoding is very complex
- choose a preferred container and videoformat
- understand lots of possible settings an encoder uses

Comments

Goals

running prototype

Week ( 12.10 - 16.10.09)

More ...

Activities

explored xuggler

Results

found 'hidden' settings for encoders in xuggler

Problems

most settings applied not to codec h.264

Comments

was ill from 13.10 to 16.10

Goals

see last week

Week ( 19.10 - 23.10.09)

Activities

worked on prototype
set up libx264 encoder to use cbr (constant bitrate encoding)
meet the given bitrate and don't exceed it
first checkin of prototype into svn

Results

prototype
- client-server program, streaming via network
- dynamic resizing of inputframes to encoder
- console in server to change and view current enviroment
- change fps while streaming
- use vbv-buffer (in ffmpeg) to guarantee the bitratelimit (in some cases, esp. low bitrates, without vbv some peaks appeared)

Problems

due to update of xuggler, 'libx264' needed other settings than default ones
- presets by ffmpeg did not work
- examples by xuggler used not cbr-mode
- bruteforce trying to find appropriate settings
in cbr mode x264 has a noticable delay
- seems that encoder collects all frames between two keyframes, encodes them at once to keep given bitrate and finally pushes them into videostream

Comments

xuggler updated to 3.3 on 21.10, so problems on setting up 'libx264' disappeared (see last week)

Goals

release saros
make weekly reports
prototype
- deal with temporary low bandwidth

Week ( 26.10 - 30.10.09)

More ...

Activities

improved prototype
releasemanager in this cycle
tested new release

Results

prototype
- extended console
- threaded server
- change resolution while streaming → problem
- introduced and second stream for controlling the transmission
  - client sends statistics about delay etc. to server
releasemanager
- collected all changes since last release
- prepared branch for testing

Problems

resizing the livestream is not possible
- used container (flv) does not support more than one videostream
- other containerformats don't seem to work
  - either not streamable (mpeg4, …)
  - or don't work w. xuggler? (mkv, ogg, …)
setting a new framerate on the fly causes a big delay on client
- reason can't be figured out, maybe is the old timebase (based on old framerate) the reason
- try to drop frames instead of changing framerate

Comments

Goals

prepare the presentation for thursday next week
- plan advance for the next weeks
- give an overview about screensharing
- details of planned implementation
  - features
  - requirements
  - integration into saros
find reason why some containerformats don't work in xuggler
prototype
- deal with temporary low bandwidth

Working stage

Less ...

Week ( 2.11 - 6.11.09)

More ...

Activities

still releasemanager, release has been deferred
worked on preparation presentation

Results

preparation presentation is done
releasemanager
- released Saros v9.10.30.r1833
- changed/expanded some steps in release instructions
- merged changes back to trunk
DPPOutReach

Problems

Comments

not done
- find reason why some containerformats don't work in xuggler

Goals

complete base architecture for screenshareapp
send some bugreports to xuggler

Week( 9.11 - 13.11.09)

More ...

Activities

worked on base architecture
meeting with Stephan and Karl on wednesday

Results

wrote usecases (ThesisDPPXIIIUseCases) and requirements
developed modules (functionality, responsibilities)
meeting
- shorten iterations to 2 weeks
- immediately integrate screensharing into saros
some bugreports for xuggler
- Issue 222
- Issue 223

Problems

Comments

this week was not really productive, i increase it!

Goals

adapt plan due to meeting
finish architecture
start integration in saros

Week ( 16.11 - 20.11.09)

More ...

Activities

preparation for implementation (requirements engineering)
worked on implementation
adapted plan

Results

architecture and modules, see ThesisDPPXIIIProgramming
started to code (see my repository)
- ported prototype to new architecture
- local testing via piped streams possible

Problems

still some open questions for architecture
- who controls when a frame will be rendered, encoder or connectionmanager?
- what is needed at clients side to initiate and handle connections?
  - need some details concerning saros, should be done this week
- how can i handle configurations?
  - will try Apache Commons Configuration
    - does it fit in eclipse?

Comments

Goals

integrate into saros
- view for clients player
- bytestreams for connection between two users

Week( 23.11 - 27.11.09)

More ...

Activities

programmed feature
- startup with saros
- get a network stream

Results

programming
- hooked into picocotainer
- tried to get network stream
entered tests 3.2 - 3.7 in testlink

Problems

is not easy to get network stream
- find a way to get one via jingle, but this seems not to be accepted → meeting tuesday

Comments

Goals

find solution for network architecture (meeting tuesday)
- let saros be packetbased, wrap a stream with them
- get a real stream via jingle/socks
- not completely my part, sandor and henning have to help me
start ui
- player
- session initiation, negotiation

Week( 30.11 - 4.12.09)

More ...

Activities

worked on streaming with saros
assisted releasemanger
tested saros for release

Results

developed framework for streambased services
- simple filetransfer-test worked

Problems

Comments

Goals

test streaming with video, finish it completely
start ui stuff

Week( 7.12 - 11.12.09)

More ...

Activities

worked at our new streaming interface
played a bit with swt and eclipse views
implemented file sending based on streaming

Results

streaming: improved it
- performance (chunkSize and maximalDelay)
- reliability, send packets same order they arrive
- introduced soft stop
UI
- created first view (canvas with simple statusbar)
- discovered swt: widgets, layouts, …

Problems

not sure whether streaming is now reliable enough… spend already a lot of time there, but sometimes strange errors occur

Comments

don't get so far to test videosharing with stream interface

Goals

integrate view into videosharing
create a StreamService for video, test it in saros

Week( 14.12 - 18.12.09)

More ...

Activities

worked on view for playing video
implemented example service for sending single files in a session
improved streaming
integration of videosharing-prototype into saros

Results

videosharing
- created branch, moved classes
- created simple view for showing video with canvas and statusbar
streaming
- can add serialized objects to metapackets
- developed improving concepts
  - streams need to be closed
  - output should block when input is not reading
file sending
- show speed in task
- cancellable

Problems

videoplayerview is swt, but images are awt: need to embed an awt canvas
sometimes streaming hangs without obvious reason, need to figure out

Comments

Goals

improve streaming (s.a.)
- esp. robustness (errorhandling, disposing)
make videosharing work in saros
- embed awt-canvas in player
- create preferences (pages)
- catch input in player's canvas, send them to host

Week( 21.12 - 25.12.09)

More ...

Ill

Week( 28.12.09 - 1.1.10)

More ...

Ill

Week( 4.1 - 8.1.10)

More ...

Ill

Week( 11.1 - 15.1.10)

More ...

Activities

finish screensharing and streaming for release
documented state of work in wiki
write paper about technical realization

Results

screensharing
- disposing
- fast awt-canvas for displaying video
- catch input in videoplayer and send to host
- pure java encoder taken from smack-demo
streaming
- errorhandling/disposing
- many fixes
paper - 50% done

Problems

alternative encoder is too slow, works only w. 1 FPS
- try to optimize

Comments

screensharing not ready for release
streaming could be ready
- christopher fixes binarychannel
- must be reviewed (and afterwards fixed)
- any feature uses it

Goals

make features ready for use for release next week
- optimize other encoder
- UI
  - players toolbar
  - resizing
  - preference dialogues
paper finished until wednesday

Week( 18.1 - 22.1.10)

More ...

Activities

writing mentioned paper
make screensharing ready for release
make streaming ready for release
reviewed some patches for release

Results

paper done
- no feedback yet
screensharing works
- UI minor bugs

Problems

seems that there's not enough to review both features
- maybe not released or only experimental?
alternative encoder
- really bad performance, only 1 fps at low resolution, even at fast machines