As I've previously written about, I am working on a Video Synthesizer plugin for VCVRack. In this post I am going to explore the Input Sources diagram, the initial ideas behind it and how I am planning on changing them.
The idea behind input sources is to provide a way to process media. In order to perform some of the image processing that'll be done later, the input sources would be critical in leveraging what images to use. For now I am limiting media to be Image, Video and Camera input. In the above diagram for Input Source, we have the most basic form for all three media sources, image, video and camera. They are their own individual modules.
- The image module is responsible for retrieving and loading an image file. It then goes ahead and makes it available for consumption through the "REF" output as shown in the drawing.
- The video module is responsible for retrieving and loading in a video file. It then process the video frame by frame and makes each frame available for consumption through the "REF" output. Unlike the image module however, the video module will have options for replaying, playing and pausing. These would be Rack params that can be modified by the user.
- The camera module on the other hand will be responsible for retrieving webcame input and making each frame available for consumption through the "REF" output. The camera module will also have params for controlling when to start capturing frames and when to stop.
All the outputs the image, video and camera modules would be float values. The float value will be a corresponding opengl texture unsigned int which will be available context wide to anyone that uses it. All three modules or components (I use them interchangeably ) will also also have a preview on the side showing the active frames.
The newer approach I am exploring at the moment however is a little different. It combines all of the input sources into one module. Because they are all already work with textures in one way or another what we need to do is to just figure out whose (Image, Video, Camera) responsible to update the texture. Once this is determined we can go ahead and determine whether to process a Video frame, use the latest Camera frame or Image. This way, we can toggle between using an Image, a Video or Camera pretty simply and have it all grouped as a single component. In addition to this grouping I think it will be beneficial to also move the preview to another component and have the main responsibility of for the Input Source only on processing inputs rather than displaying them as well.
There are many other beneficits to the newer approach. For one there is less modules to throw arround the Rack board. This will mean less wirres moving accross the board when I have a component that wants to be able to retrieve a texture from an Image module but then decides to retrieve it from a Video module vice versa. It kinda also makes sense to group them together because they are not entirely different in functionality.