Skip to content

Sync pg to solr

Schedule for syncing data from Postgres to Solr (streaming).

sync_pg_to_solr_task()

Sync data from Postgres to Solr every 15 minutes, streaming & batched.

Source code in gso/schedules/sync_pg_to_solr.py 
10
11
12
13
14
15
16
17
18
@shared_task
@scheduler(CronScheduleConfig(name="Sync data from Postgres to Solr", minute="*/15"))
def sync_pg_to_solr_task() -> None:
    """Sync data from Postgres to Solr every 15 minutes, streaming & batched."""
    indexing_batch_size = load_oss_params().SOLR.indexing_batch_size

    ensure_solr_copy_field()
    docs_stream = stream_all_data()
    post_to_solr(docs_stream, batch_size=indexing_batch_size)