Project Overview
Friday is an advanced, multimodal voice assistant that bridges the gap between natural voice reasoning and OS-level execution. By integrating LiveKit WebRTC channels with Google Gemini Realtime APIs, the assistant processes voice instructions with sub-second latency, executing local commands through a secured tool-calling architecture.
Product Demonstration
Problem & Motivation
The Problem
Traditional automation requires writing custom scripts or using rigid hotkeys. Non-technical tasks (like organizing files or checking build logs) take active context-switching. Friday resolves this by offering a zero-interface voice gateway that interacts with the operating system securely.
The Motivation
Most voice assistants are walled gardens confined to answering generic web questions. The motivation behind Friday was to build a truly useful agentic helper that has full access to the local machine, allowing hands-free developer workflows, automated directory setups, and instant system actions through natural conversation.
System Architecture
Friday utilizes a client-server WebRTC setup. The frontend Next.js application connects to the LiveKit server to capture user audio. The backend agent node (Python) listens to the audio stream, pipes it to the Gemini Realtime WebSocket, and handles tool-call responses dynamically (e.g. running shell scripts, editing files).
Secured tool-calling framework (automation.py & tools.py) implementing strict file-range and command validation sandboxes.
Real-time audio streaming pipeline using LiveKit's WebRTC infrastructure to minimize voice loopback latency under 600ms.
Context persistence layer powered by mem0, allowing the assistant to remember project names, developer preferences, and user history across restarts.
Custom MCP (Model Context Protocol) client integration allowing Friday to connect to local development tools and databases.
Key Features & Capabilities
Capability Explorer
Friday is integrated directly with the workstation's local environment through a registry of 45+ system, mouse, keyboard, search, and device controller scripts. Click any category to inspect its integrated tools.
Information & Search
Retrieves weather reports, searches the web, or queries YouTube paths.
get_weather()search_web()get_local_time()search_youtube()Desktop Automation
Launches local applications, controls dark mode, and handles power loops.
open_app()set_dark_mode()shutdown_system()restart_system()lock_screen()System Controls
Monitors battery specs, triggers volumes, and routes mic streams.
unlock_screen()battery_status()adjust_volume()mute_volume()unmute_volume()mute_microphone()unmute_microphone()File System
Audits folder structures, writes/reads file segments, and handles directories.
clean_temp_files()open_folder()search_file()open_file()create_directory()write_file()read_file()Mouse & Keyboard Control
Positions cursor coords, triggers keystrokes, and types string logs.
move_mouse()click_mouse()cursor_cordinate()type_text()press_keys()scroll_mouse()press_or_click()switch_window()Browser & Media
Loads web pages, manages media playback, and opens settings panels.
play_music()open_website()open_settings_page()AI Generation
Drafts detailed copy plans and generates image assets locally.
image_genrt()generate_content()Scheduling & Productivity
Triggers notification alarms, logs focus states, and tracks processes.
add_reminder()get_due_reminders()monitor_app_focus()Mobile Automation
Bridges WhatsApp networks, unlocks screens, and dials phone lines.
unlock_phone()open_mobile_app()close_mobile_app()auto_dial_contact()send_whatsapp_message()get_screen_lock_state()Agent Core Execution Architecture
Input
User triggers Voice WebRTC stream
Parsing
Intent parameters parsed via Gemini Realtime
Planner
Compiles dependency execution tree paths
Tool Resolve
Locates function schemes inside registry
Execution
Runs command scripts on local OS
Observation
Inspects output log results and codes
Response
TTS synthesizes response voice feedback
Engineering Challenges
Voice Audio Latency Over WebRTC
Standard HTTP request loops for audio upload and text-to-speech feedback resulted in average response latency exceeding 3 seconds, making fluid voice conversation impossible.
Replaced API polling with a persistent bi-directional WebRTC connection via LiveKit. The audio is streamed chunk-by-chunk and sent directly to Gemini's Realtime WebSocket interface, reducing complete round-trip latency to roughly 650ms.
Development Timeline
Conceived Friday Voice Agent system architecture.
Wired LiveKit RTC audio capture and Gemini Realtime WebSocket hooks.
Integrated mem0 memory storage and completed standard local tools suite.
Lessons Learned
- Real-time state orchestration is complex; keeping voice buffers synchronized requires robust WebSocket handling.
- Securing local OS agents is critical; arbitrary command execution must always be restricted by whitelist schemas.
Future Improvements
- Integrate local vector-embedding searches for on-device codebase comprehension.
- Support fully offline operation by routing voice pipelines to local Whisper and LLaMA instances.