Keep your Data Flowing

0 of 5 lessons complete (0%)

Lesson 2: Building the SQL Queries

You don’t have access to this lesson

Please register or sign in to access the course content.

The following lesson content assumes that you have read access to the FME Flow database, and publish permissions on ArcGIS Server and ArcGIS Monitor.

Warning: The rest of this article covers accessing the FME Flow database directly, whether it’s PostgreSQL or SQL Server. This is risky territory (and probably not supported). You don’t want to edit the FME Flow tables, so proceed with caution and use a separate read-only account against the FME schema whenever possible. Some of the queries also need to exist as views in the FME Flow database to work as query layers in ArcGIS, so you’ll need create-view permissions as well.

These queries/dashboards have been tested on:

  • An FME Flow 2025.1 instance using a PostgreSQL 16 database, paired with ArcGIS Enterprise 11.5
  • An FME Flow 2026.2 instance using a SQL Server 2025 database, paired with ArcGIS Enterprise 12.1
  • ArcGIS Monitor 2026.0 (the ability to execute database queries was added in Monitor 2026)

We’ll be working with the following tables in the FME Flow database. In a default installation, FME Flow installs its own version of PostgreSQL. It can also be configured to work with SQL Server and Oracle.

Job History & Status:

  • fme_jobs: Data about jobs that are currently running or queued.
  • fme_job_history: Information about jobs that have been cancelled, failed, or completed.
  • fme_job_status: A lookup table for the status IDs used in the job tables.

Nodes & Engines:

  • fme_engines: Information about the engines that have registered into the server pool.
  • fme_core_node: Information about the cores that have been installed and are connected to the cluster.

Now that we have the tables described, let’s go over the queries that we are going to be using.

The example queries below are for a PostgreSQL database. SQL Server versions can be found in our GitHub repository.

FME Observability GitHub Repo

All Jobs Last 48 Hours:

This query combines currently running jobs with job history to create a single view for use in the ArcGIS Dashboard. It selects jobs completed, cancelled, or failed within the last 48 hours. A 48-hour window was chosen because it’s enough time to spot a trend without loading the dashboard with more data than it needs.

SELECT *
FROM (SELECT jh.job_id,
             jh.date_queued,
             jh.date_started,
             jh.date_finished,
             (jh.elapsedtime / 1000)::bigint AS runtime,
             jh.tag,
             jh.repository_name,
             jh.workspace_name,
             CASE --clean up the machine names
              WHEN jh.engine_instance LIKE '%.%'
                THEN split_part(jh.engine_instance, '.', 1)
              ELSE split_part(jh.engine_instance, '_', 1)
             END                              AS engine_instance,
             js.job_status                   AS status,
             CASE --Create a field to aid with sorting running and queued jobs.
              WHEN date_started IS NULL AND date_finished IS NULL THEN 1 -- queued
              WHEN date_started IS NOT NULL AND date_finished IS NULL THEN 2 -- running
              ELSE 3 -- canceled, finished, etc
             END                             AS sorder,
             ua.username
      FROM public.fme_job_history AS jh
               INNER JOIN public.fme_job_status AS js
                          ON jh.job_status = js.id
               INNER JOIN public.fme_useraccount AS ua
                          ON jh.useraccount_id = ua.useraccount_id
      WHERE jh.date_finished >= NOW() - INTERVAL '48 hours'

      UNION ALL

      SELECT j.job_id,
             j.date_queued,
             j.date_started,
             j.date_finished,
             (j.elapsedtime / 1000)::bigint AS runtime,
             j.tag,
             j.repository_name,
             j.workspace_name,
             CASE --clean up the machine names
              WHEN j.engine_instance LIKE '%.%'
                THEN split_part(j.engine_instance, '.', 1)
              ELSE split_part(j.engine_instance, '_', 1)
             END                             AS engine_instance,
             js.job_status                  AS status,
             CASE --Create a field to aid with sorting running and queued jobs.
              WHEN date_started IS NULL AND date_finished IS NULL THEN 1 -- queued
              WHEN date_started IS NOT NULL AND date_finished IS NULL THEN 2 -- running
              ELSE 3 -- canceled, finished, etc
             END                            AS sorder,
             ua.username
      FROM public.fme_jobs AS j
               INNER JOIN public.fme_job_status AS js
                          ON j.job_status = js.id
               INNER JOIN public.fme_useraccount AS ua
                          ON j.useraccount_id = ua.useraccount_id) AS all_jobs
ORDER BY sorder, date_queued DESC;

Save this query as a view in your FME Flow database and provide your ArcGIS Server account read access to the view.

FME Nodes & Engines:

This query pulls the nodes and engines currently registered in the FME Flow database and combines them into a single view. It answers the earlier question: are all the engines registered in the cluster? In a high-availability deployment, this is how you’d catch an Engine (or even a Core) that’s silently dropped out.

SELECT row_number() OVER (ORDER BY name) AS id,
       name,
       hostname,
       engine_manager_node_name,
       role,
       job_id,
       is_leader,
       numstandardengines
FROM (SELECT
             CASE
                 WHEN fme_engines.name ~ '^\d+\.\d+\.\d+\.\d+$'
                     THEN fme_engines.name
                 WHEN fme_engines.name LIKE '%.%'
                     THEN split_part(fme_engines.name, '.', 1)::character varying
                 ELSE split_part(fme_engines.name, '_', 1)::character varying
                 END        AS name,
             CASE
                 WHEN fme_engines.hostname ~ '^\d+\.\d+\.\d+\.\d+$'
                     THEN fme_engines.hostname
                 WHEN fme_engines.hostname LIKE '%.%'
                     THEN split_part(fme_engines.hostname, '.', 1)::character varying
                 ELSE split_part(fme_engines.hostname, '_', 1)::character varying
                 END        AS hostname,
             CASE
                 WHEN fme_engines.engine_manager_node_name ~ '^\d+\.\d+\.\d+\.\d+$'
                     THEN fme_engines.engine_manager_node_name
                 WHEN fme_engines.engine_manager_node_name LIKE '%.%'
                     THEN split_part(fme_engines.engine_manager_node_name, '.', 1)::character varying
                 ELSE split_part(fme_engines.engine_manager_node_name, '_', 1)::character varying
                 END        AS engine_manager_node_name,
             'Engine'::text AS role,
             fme_engines.job_id,
             NULL::boolean  AS is_leader,
             NULL::integer  AS numstandardengines
      FROM fme_engines
      WHERE fme_engines.type <> 2

      UNION ALL

      SELECT
             CASE
                 WHEN fme_core_node.name ~ '^\d+\.\d+\.\d+\.\d+$'
                     THEN fme_core_node.name
                 WHEN fme_core_node.name LIKE '%.%'
                     THEN split_part(fme_core_node.name, '.', 1)::character varying
                 ELSE split_part(fme_core_node.name, '_', 1)::character varying
                 END                 AS name,
             CASE
                 WHEN fme_core_node.name ~ '^\d+\.\d+\.\d+\.\d+$'
                     THEN fme_core_node.name
                 WHEN fme_core_node.name LIKE '%.%'
                     THEN split_part(fme_core_node.name, '.', 1)::character varying
                 ELSE split_part(fme_core_node.name, '_', 1)::character varying
                 END                 AS hostname,
             'NA'::character varying AS engine_manager_node_name,
             'Core'::text            AS role,
             NULL::integer           AS job_id,
             fme_core_node.is_leader,
             fme_core_node.numstandardengines
      FROM fme_core_node

      UNION ALL

      SELECT
             CASE
                 WHEN fme_remote_engine_conn.name ~ '^\d+\.\d+\.\d+\.\d+$'
                     THEN fme_remote_engine_conn.name
                 WHEN fme_remote_engine_conn.name LIKE '%.%'
                     THEN split_part(fme_remote_engine_conn.name, '.', 1)::character varying
                 ELSE split_part(fme_remote_engine_conn.name, '_', 1)::character varying
                 END                 AS name,
             CASE
                 WHEN fme_remote_engine_conn.name ~ '^\d+\.\d+\.\d+\.\d+$'
                     THEN fme_remote_engine_conn.name
                 WHEN fme_remote_engine_conn.name LIKE '%.%'
                     THEN split_part(fme_remote_engine_conn.name, '.', 1)::character varying
                 ELSE split_part(fme_remote_engine_conn.name, '_', 1)::character varying
                 END                 AS hostname,
             'NA'::character varying AS engine_manager_node_name,
             'Remote Engine'::text   AS role,
             NULL::integer           AS job_id,
             NULL::boolean           AS is_leader,
             fme_remote_engine_conn.numstandardengines
      FROM fme_remote_engine_conn) t;

Save this query as a view in your FME Flow database and provide your ArcGIS Server account read access to the view.