Shape Reporting Views and Retention Checks Before a Streamlit Dashboard Goes Live

Suggested cover: a live operations screen where one worker aggregates traffic history while another keeps reading video and feeding the database.

Background loops are where a script stops being a demo and becomes a system

By the time this section begins, the code has already shown us how one frame becomes one database-ready record. The next question is harder and more operational: how does that work continue without a person pressing a button for every step. This is where the script grows into a living process. It introduces a loop that continuously rolls current data into historical aggregates and another loop that keeps the video pipeline alive.

There is a lot of engineering texture in that shift. Words like daemon thread, aggregation cadence and continuous ingestion are not decorative labels here. They describe the social reality of the code. One part of the file is concerned with seeing vehicles now, another is concerned with preserving an hourly memory of what just happened, and both have to coexist without making the Streamlit layer feel frozen.

These loops are short, but short loops can still carry a surprising amount of responsibility. They decide the rhythm of data freshness, the survival behavior after transient errors and the overall tone of the runtime under constant motion.

Pipeline term	Meaning	Why it matters here
Aggregation cadence	The interval at which recent operational data is rolled into historical form.	It determines how quickly the dashboard stops showing raw turbulence and starts showing reusable history.
Daemon thread	A background worker that runs without becoming the main control path of the application.	It lets live processing continue while the UI remains responsive.
Loop resilience	The ability of a repeated task to keep functioning through ordinary transient failures.	It prevents the whole runtime from feeling brittle the first time a connection or read stumbles.

The historical rollup loop begins by claiming a steady rhythm

def run_streaming_pipeline():
    while True:
        try:
            conn = psycopg2.connect(**db_config)
            cur = conn.cursor()
            cur.execute("""

The opening is spare on purpose: one endless loop, one connection, one cursor and then an aggregation query that takes over the real work.

The complexity does not come from fancy syntax. It comes from the promise that this little loop is expected to keep showing up, keep connecting and keep translating the recent operational table into a calmer historical layer.

The aggregation query keeps the history table refreshed in place

                INSERT INTO mart.traffic_history (report_hour, camera_id, vehicle_type, avg_speed, intensity, density)
                SELECT date_trunc('hour', last_update), camera_id, vehicle_type, AVG(avg_speed), COUNT(*), AVG(count)
                FROM mart.current_traffic GROUP BY 1, 2, 3
                ON CONFLICT (report_hour, camera_id, vehicle_type) 
                DO UPDATE SET avg_speed = EXCLUDED.avg_speed, intensity = EXCLUDED.intensity, density = EXCLUDED.density;
            """)

This is the point where fleeting live detections become a reusable hourly history that later analytics can revisit.

The query is compact, but it carries the emotional center of the pipeline. It says that the system values not only current state, but also a rolling memory of recent conditions. That one choice is what allows later charts, KPI summaries and forecasts to exist without re-reading the volatile table every time.

The loop closes, sleeps and then hands off to video processing

            conn.commit()
            cur.close()
            conn.close()
            time.sleep(5)
        except:
            time.sleep(5)

def process_video():
    model = YOLO(MODEL_NAME)
    cap = cv2.VideoCapture("test.MP4")
    track_history = {}

The end of one loop folds directly into the setup of another, which makes the file read like a relay rather than a bag of unrelated functions.

The resilience is intentionally modest. On failure the loop simply waits and tries again. Then the next function prepares the model and capture object for the continuous vision path. It is not theatrical error handling, but it is enough to make the runtime feel patient instead of brittle.

The video worker and bootstrap threads complete the live handoff

    try:
        font = ImageFont.truetype("arial.ttf", 20)
    except:
        font = ImageFont.load_default()
    while cap.isOpened():
        conn = psycopg2.connect(**db_config)
        cur = conn.cursor()
        frame = get_raw_frame(cap)
        if frame is not None:
            db_data, frame = process_and_draw(frame, model, track_history, font)
            if db_data:
                push_to_database(cur, conn, db_data)
        cur.close()
        conn.close()

if 'video_thread_started' not in st.session_state:
    threading.Thread(target=process_video, daemon=True).start()
    threading.Thread(target=run_streaming_pipeline, daemon=True).start()
    st.session_state.video_thread_started = True

This closing slice makes the runtime topology explicit: one thread keeps the video pipeline alive while another keeps the history rollup fresh.

That is where the whole script starts to feel assembled rather than merely written. The file is no longer just a collection of capabilities. It now contains a live arrangement that explains why the dashboard layer later on will have both current and historical data to work with.

• This section concentrates on the background workers that keep ingestion and aggregation moving in parallel.
• The code remains chronological, so the handoff from recurring updates to startup threads is easy to follow.
• The article text frames the runtime as a coordinated system rather than a pile of disconnected loops.

A script becomes a system when its loops start cooperating on time instead of merely existing in the same file.