Virtual Party Space Devlog #16: AWS architecture with docker-compose

Today I finally stopped procrastinating and started seriously looking into how to build this on AWS. (But also fixed a bunch of bugs!)

Log

• looked into memory leaks
  • initial tests show that memory growth doesn’t present over short timeframes
• added userId to events to allow for multiple-player movement
• modified depth layers so player avatars always drawn in front of scenery at “same” depth (using step-2 as considered before)
  • still looks weird when player going behind things, but moving in front of buildings now looks right

Bug fixes

• took forever to figure out mute bug!
  • calling toggleAudioMuted in conferenceJoined event handler was the problem
    • conferenceJoined event is dispatched before async setAudioInput completes, so toggle would find no input and create a new one, which would then get destroyed when the initial audio input tried to complete.
    • always saw 2 adds, 2 removes, so maybe the async calls stepped on each other?
    • left with no audio track, new audio track would then be created on hitting mute toggle or switching inputs
  • combined with Calla’s not-great idea to use “toggle” instead of any way of setting directly
    • isAudioMuted was always null so would always toggle, regardless of “set” value
    • even after patching that, initial state is always null, so leads to different behaviour depending if audio track exists or not
  • no easy way to create a track muted
    • startAudioMuted in Jitsi doesn’t seem to do anything?
    • also tried startMutedPolicy, also doesn’t seem to do anything
      • maybe look into these later, but Jitsi is complicated.
• reverted a change to Calla and then discovered it’s required to have the “None” option in audio/video dropdowns
  • redid fix and committed this time
  • still on “fixes” branch of official repo, haven’t recalled how to switch origins
• remote user avatars now properly load to that user’s reported position
  • addParticipant is getting sloppy, might want to break this out into somewhere else
• figured out how to properly add user avatars & positions for all users!
  • requires sending custom Jitsi messages. Calla doesn’t have an interface for this, so now I’ve polluted my conference code with hard-coded Jitsi references

CANNOT PUT OFF AWS ANYMORE

• A recommended Fargate and I guess I see it, going to try tutorials in that direction
• installed ecs cli and upgraded aws-cli
  • ****TODO: set up AWS profile
• Eventually get an Elastic IP
  • cost <$5 for a month
  • add CNAME at domain registrar well in advance to get domain name

Using docker-compose

• It’s possible to use docker-compose with ECS
  • https://docs.docker.com/cloud/ecs-integration/
  • this seems to avoid the need to create cloudformation templates and just turns your docker-compose.yml into cloudformation
    • can use docker compose convert to just generate the cloudformation and manually tweak
  • can reuse much of the existing jitsi docker project, just update if I need to do something custom with jvb
    • I think I’m getting the picture that despite the visualization in Docker Desktop, all the different containers are actually entirely separate
  • can also test AWS context locally https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/
  • need to add features like:
    • resources (max CPU and memory per service)
    • x-aws-* parameters
• Can specify a .env file for docker-compose https://docs.aws.amazon.com/AmazonECS/latest/developerguide/taskdef-envfiles.html
  • unfortunately this might not work for the docker binary, which is required for ECS features
    • .env file support added a couple weeks ago, might work? Installing new Docker Desktop version
• ****TODO: figure out SSL certificates
• ****TODO: pull passwords out of .env and put them into secrets manager
  • learn how secrets work I guess

Other considerations

• I think the only thing the “web” docker container is used for is proxying http-bind on to xmpp.meet.jitsi:5280
  • could ditch “web” altogether, and replace it with a container of my app?
• my code is getting complicated enough that I’m starting to break features when adding a new feature, probably need tests
  • unfortunately most of my test needs are integration/e2e, not sure how to test ie jitsi connection events
    • a mock of lib-jitsi-meet would take so long to write

About this series

Back in mid-December I started an ambitious project to create a custom platform for a virtual birthday party in February. I kept notes on my progress, both for personal reference and to turn into a series of blog posts. It quickly became apparent that I did not have time to both do the project and blog about the project. I have retroactively decided to post my raw notes as a dev log.