Window¶

The "window" transform calculates values for every input row from a frame of rows in its partition. It keeps input rows and preserves their observed output order. Sorting affects calculations only: "window" is not a sorting transform.

Parameters¶

as

Type: array

Output field names. A missing or null entry uses the operation and field name joined with an underscore, such as sum_score. Operations without a field use only the operation name, such as rank.

description

Type: string

A description of the transform step. Can be used for documentation and agent context.

fields

Type: array

Input fields for operations that use a field. Use null for operations such as rank and count that do not use one.

frame

Type: array

Inclusive offsets from the current sorted row that define the window. null leaves the corresponding side unbounded.

Default value: [null, 0]

groupby

Type: array

Fields that divide the input into independent window partitions.

ignorePeers

Type: boolean

Use row offsets without expanding frame boundaries to include sorted rows with equal values.

Default value: false

ops Required

Type: array

Window and aggregate operations to calculate. Entries align with fields, params, and as.

params

Type: array

Optional operation parameters. lag and lead use an offset, while ntile and nth_value use a positive integer.

sort

Type: CompareParams

Fields used to sort rows before window functions are calculated. Without sorting, rows retain their input order and no rows are peers.

Partitions and order¶

groupby defines independent partitions. For example, "groupby": ["sample"] ensures that a sample's lead value never comes from a different sample. sort defines the calculation order within each partition. Without sort, observed input order defines the window and rows are not peers.

The following configuration calculates nextValue from the next row in each sample after sorting by position:

{
  "type": "window",
  "groupby": ["sample"],
  "sort": { "field": "position" },
  "ops": ["lead"],
  "fields": ["value"],
  "as": ["nextValue"]
}

The resulting rows still propagate in their original order. This matters when a window calculation is followed by marks or transforms that should retain the source order.

Frames and peers¶

The frame contains inclusive offsets relative to the current sorted row. [0, 0] contains only the current row, [-1, 0] contains the preceding and current rows, and [null, 0] contains every preceding row and the current row. null makes a bound unbounded. [null, 0] is the default.

With a sort order, a frame boundary includes every tied row unless ignorePeers is true. This makes an aggregate such as a running sum agree for tied sort values. ignorePeers: true instead applies exact row offsets.

lag and lead use the sorted partition order but do not use the frame. Their offset defaults to one. ntile and nth_value require a positive integer in the aligned params entry.

Operations and output fields¶

ops, fields, params, and as are aligned by index. count and ranking operations have no field entry. If an as entry is omitted or null, the output name is the operation name followed by _<field> when applicable.

Operation group	Operations
Row order and ranking	`row_number`, `rank`, `dense_rank`, `percent_rank`, `cume_dist`, `ntile`
Relative values	`lag`, `lead`, `prev_value`, `next_value`
Frame values	`first_value`, `last_value`, `nth_value`
Aggregates	`count`, `valid`, `sum`, `min`, `max`, `mean`, `q1`, `median`, `q3`, `variance`

prev_value and next_value return the nearest non-null value at or before, or at or after, the current row. Aggregate operations ignore null, empty-string, and NaN input values. Empty aggregate frames produce the corresponding empty aggregate result; frame-value operations return null when their requested row is outside the frame.

Batches¶

The transform calculates each data-flow batch independently. A window never crosses a file or facet batch boundary.

Example¶

The arrows show lead values by pointing from each row to its next row. Filled blue points are the input values. Open points are the two-row moving mean from the frame [-1, 0].

{
  "description": "Window transform showing lead arrows and a two-row moving mean.",

  "data": {
    "values": [
      { "position": 1, "value": 2 },
      { "position": 2, "value": 5 },
      { "position": 3, "value": 3 },
      { "position": 4, "value": 7 },
      { "position": 5, "value": 6 }
    ]
  },

  "transform": [
    {
      "type": "window",
      "sort": { "field": "position" },
      "ops": ["lead", "lead", "mean"],
      "fields": ["position", "value", "value"],
      "as": ["nextPosition", "nextValue", "movingMean"],
      "frame": [-1, 0]
    }
  ],

  "encoding": {
    "x": { "field": "position", "type": "quantitative" },
    "y": { "field": "value", "type": "quantitative" }
  },

  "layer": [
    {
      "mark": { "type": "point", "filled": true },
      "encoding": { "color": { "value": "#4C78A8" } }
    },
    {
      "mark": "point",
      "encoding": {
        "y": { "field": "movingMean", "type": "quantitative" },
        "color": { "value": "black" }
      }
    },
    {
      "transform": [
        { "type": "filter", "expr": "isValid(datum.nextPosition)" }
      ],
      "mark": {
        "type": "arrow",
        "size": 1,
        "headWidth": 15
      },
      "encoding": {
        "x2": { "field": "nextPosition" },
        "y2": { "field": "nextValue" },
        "color": { "value": "#A0A0A0" }
      }
    }
  ]
}

Six-frame FASTA translation¶

The six-frame FASTA translation example below sorts flattened bases by genomic position and uses four lead operations to gather the next two bases and complements for every row.

{
  "description": [
    "Indexed FASTA Six-Frame Translation",
    "Translates a visible hg38 FASTA interval in three forward and three reverse reading frames using generic flattenSequence, lookup, window, and formula transforms.",
    "Data source: UCSC hg38 / GRCh38 reference FASTA (`goldenPath/hg38/bigZips/latest/hg38.fa.gz`). Terms: UCSC downloadable data files are freely reusable."
  ],

  "assembly": "hg38",

  "name": "indexed-fasta-six-frame-translation",

  "scales": {
    "x": {
      "domain": [
        { "chrom": "chr7", "pos": 20395929 },
        { "chrom": "chr7", "pos": 20395976 }
      ]
    }
  },

  "datasets": {
    "nucleotideComplements": [
      { "base": "A", "complement": "T" },
      { "base": "C", "complement": "G" },
      { "base": "G", "complement": "C" },
      { "base": "T", "complement": "A" },
      { "base": "N", "complement": "N" }
    ],
    "geneticCode": [
      { "codon": "TTT", "aminoAcid": "F", "kind": "other" },
      { "codon": "TTC", "aminoAcid": "F", "kind": "other" },
      { "codon": "TTA", "aminoAcid": "L", "kind": "other" },
      { "codon": "TTG", "aminoAcid": "L", "kind": "other" },
      { "codon": "TCT", "aminoAcid": "S", "kind": "other" },
      { "codon": "TCC", "aminoAcid": "S", "kind": "other" },
      { "codon": "TCA", "aminoAcid": "S", "kind": "other" },
      { "codon": "TCG", "aminoAcid": "S", "kind": "other" },
      { "codon": "TAT", "aminoAcid": "Y", "kind": "other" },
      { "codon": "TAC", "aminoAcid": "Y", "kind": "other" },
      { "codon": "TAA", "aminoAcid": "STOP", "kind": "stop" },
      { "codon": "TAG", "aminoAcid": "STOP", "kind": "stop" },
      { "codon": "TGT", "aminoAcid": "C", "kind": "other" },
      { "codon": "TGC", "aminoAcid": "C", "kind": "other" },
      { "codon": "TGA", "aminoAcid": "STOP", "kind": "stop" },
      { "codon": "TGG", "aminoAcid": "W", "kind": "other" },
      { "codon": "CTT", "aminoAcid": "L", "kind": "other" },
      { "codon": "CTC", "aminoAcid": "L", "kind": "other" },
      { "codon": "CTA", "aminoAcid": "L", "kind": "other" },
      { "codon": "CTG", "aminoAcid": "L", "kind": "other" },
      { "codon": "CCT", "aminoAcid": "P", "kind": "other" },
      { "codon": "CCC", "aminoAcid": "P", "kind": "other" },
      { "codon": "CCA", "aminoAcid": "P", "kind": "other" },
      { "codon": "CCG", "aminoAcid": "P", "kind": "other" },
      { "codon": "CAT", "aminoAcid": "H", "kind": "other" },
      { "codon": "CAC", "aminoAcid": "H", "kind": "other" },
      { "codon": "CAA", "aminoAcid": "Q", "kind": "other" },
      { "codon": "CAG", "aminoAcid": "Q", "kind": "other" },
      { "codon": "CGT", "aminoAcid": "R", "kind": "other" },
      { "codon": "CGC", "aminoAcid": "R", "kind": "other" },
      { "codon": "CGA", "aminoAcid": "R", "kind": "other" },
      { "codon": "CGG", "aminoAcid": "R", "kind": "other" },
      { "codon": "ATT", "aminoAcid": "I", "kind": "other" },
      { "codon": "ATC", "aminoAcid": "I", "kind": "other" },
      { "codon": "ATA", "aminoAcid": "I", "kind": "other" },
      { "codon": "ATG", "aminoAcid": "M", "kind": "start" },
      { "codon": "ACT", "aminoAcid": "T", "kind": "other" },
      { "codon": "ACC", "aminoAcid": "T", "kind": "other" },
      { "codon": "ACA", "aminoAcid": "T", "kind": "other" },
      { "codon": "ACG", "aminoAcid": "T", "kind": "other" },
      { "codon": "AAT", "aminoAcid": "N", "kind": "other" },
      { "codon": "AAC", "aminoAcid": "N", "kind": "other" },
      { "codon": "AAA", "aminoAcid": "K", "kind": "other" },
      { "codon": "AAG", "aminoAcid": "K", "kind": "other" },
      { "codon": "AGT", "aminoAcid": "S", "kind": "other" },
      { "codon": "AGC", "aminoAcid": "S", "kind": "other" },
      { "codon": "AGA", "aminoAcid": "R", "kind": "other" },
      { "codon": "AGG", "aminoAcid": "R", "kind": "other" },
      { "codon": "GTT", "aminoAcid": "V", "kind": "other" },
      { "codon": "GTC", "aminoAcid": "V", "kind": "other" },
      { "codon": "GTA", "aminoAcid": "V", "kind": "other" },
      { "codon": "GTG", "aminoAcid": "V", "kind": "other" },
      { "codon": "GCT", "aminoAcid": "A", "kind": "other" },
      { "codon": "GCC", "aminoAcid": "A", "kind": "other" },
      { "codon": "GCA", "aminoAcid": "A", "kind": "other" },
      { "codon": "GCG", "aminoAcid": "A", "kind": "other" },
      { "codon": "GAT", "aminoAcid": "D", "kind": "other" },
      { "codon": "GAC", "aminoAcid": "D", "kind": "other" },
      { "codon": "GAA", "aminoAcid": "E", "kind": "other" },
      { "codon": "GAG", "aminoAcid": "E", "kind": "other" },
      { "codon": "GGT", "aminoAcid": "G", "kind": "other" },
      { "codon": "GGC", "aminoAcid": "G", "kind": "other" },
      { "codon": "GGA", "aminoAcid": "G", "kind": "other" },
      { "codon": "GGG", "aminoAcid": "G", "kind": "other" }
    ]
  },

  "resolve": { "axis": { "x": "shared" } },

  "spacing": 5,

  "data": {
    "lazy": {
      "type": "indexedFasta",
      "url": "https://data.genomespy.app/genomes/hg38/hg38.fa"
    }
  },

  "transform": [
    {
      "type": "flattenSequence",
      "field": "sequence",
      "as": ["rawPos", "base"]
    },
    {
      "type": "formula",
      "expr": "datum.start + datum.rawPos",
      "as": "pos"
    }
  ],

  "vconcat": [
    {
      "name": "reference-bases",

      "height": 20,

      "encoding": {
        "x": { "chrom": "chrom", "pos": "pos", "type": "locus" },
        "color": {
          "field": "base",
          "type": "nominal",
          "scale": {
            "domain": ["A", "C", "T", "G", "a", "c", "t", "g", "N"],
            "range": [
              "#7BD56C",
              "#FF9B9B",
              "#86BBF1",
              "#FFC56C",
              "#7BD56C",
              "#FF9B9B",
              "#86BBF1",
              "#FFC56C",
              "#E0E0E0"
            ]
          },
          "legend": null
        }
      },
      "layer": [
        {
          "name": "reference-base-background",
          "mark": { "type": "rect", "tooltip": null }
        },
        {
          "name": "reference-base-labels",
          "mark": {
            "type": "text",
            "size": 13,
            "fitToBand": true,
            "paddingX": 1.5,
            "paddingY": 1,
            "opacity": 0.7,
            "flushX": false,
            "tooltip": null
          },
          "encoding": {
            "color": { "value": "black" },
            "text": { "field": "base" }
          }
        }
      ]
    },
    {
      "name": "translation",

      "height": { "step": 17 },
      "view": { "stroke": "#c0c0c0" },

      "encoding": {
        "y": {
          "field": "lane",
          "type": "ordinal",
          "scale": {
            "domain": [
              "forward 2",
              "forward 1",
              "forward 0",
              "reverse 0",
              "reverse 1",
              "reverse 2"
            ]
          },
          "axis": { "title": null }
        },
        "x": {
          "chrom": "chrom",
          "pos": "pos",
          "type": "locus",
          "band": 0
        },
        "x2": { "chrom": "chrom", "pos": "end", "band": 0 }
      },

      "transform": [
        { "type": "formula", "expr": "upper(datum.base)", "as": "base" },
        {
          "type": "lookup",
          "from": { "name": "nucleotideComplements" },
          "key": "base",
          "values": ["complement"],
          "default": "N"
        },
        {
          "type": "window",
          "sort": { "field": "pos" },
          "ops": ["lead", "lead", "lead", "lead"],
          "fields": ["base", "base", "complement", "complement"],
          "params": [1, 2, 1, 2],
          "as": ["base1", "base2", "complement1", "complement2"]
        },
        {
          "type": "filter",
          "expr": "isValid(datum.base2) && isValid(datum.complement2)"
        },
        { "type": "formula", "expr": "datum.pos + 3", "as": "end" }
      ],

      "templates": {
        "amino-acid-translation": {
          "params": [{ "name": "strand", "value": "forward" }],

          "transform": [
            {
              "type": "formula",
              "expr": "strand === 'reverse' ? datum.complement2 + datum.complement1 + datum.complement : datum.base + datum.base1 + datum.base2",
              "as": "codon"
            },
            {
              "type": "lookup",
              "from": { "name": "geneticCode" },
              "key": "codon",
              "default": "?"
            },
            {
              "type": "formula",
              "expr": "strand + ' ' + (datum.pos % 3)",
              "as": "lane"
            }
          ],

          "layer": [
            {
              "name": "amino-acids",
              "mark": {
                "type": "arrow",
                "style": "arrow-block-notch",
                "stroke": "#C0C0C0",
                "headAngle": 65,
                "strokeWidth": {
                  "expr": "1.0 - smoothstep(0.2, 1, span(domain('x')) / width)"
                },
                "tooltip": null
              },
              "encoding": {
                "direction": { "value": { "expr": "strand" } },
                "color": {
                  "field": "kind",
                  "type": "nominal",
                  "scale": {
                    "domain": ["start", "stop", "other", "?"],
                    "range": ["#40B050", "#F06060", "#F8F8F8", "#E0E0E0"]
                  },
                  "legend": null
                }
              }
            },
            {
              "name": "amino-acid-labels",
              "mark": {
                "type": "text",
                "size": 12,
                "paddingX": 1.5,
                "tooltip": null
              },
              "encoding": {
                "color": { "value": "black" },
                "text": { "field": "aminoAcid" }
              }
            }
          ]
        }
      },

      "layer": [
        {
          "import": { "template": "amino-acid-translation" },
          "name": "forward-translation",
          "params": { "strand": "forward" }
        },
        {
          "import": { "template": "amino-acid-translation" },
          "name": "reverse-translation",
          "params": { "strand": "reverse" }
        }
      ]
    }
  ]
}

See Indexed FASTA Six-Frame Translation for a more detailed description of the example and its data sources.