Data Management, Transforms & Workflow
Data management, transforms and workflow procedures in Statios: dataset loading and multi-dataset workflows, direct cell editing with undo/redo, spreadsheet-style formulas for derived variables, variable metadata configuration (types, labels, measurement levels), data transformations (compute, recode, categorize, standardize, filter, lag/lead, dummy), data manipulation (merge, reshape, aggregate, collapse, append, sort, subset, deduplicate, rank, transpose, split/concat columns, pivot), data-to-table formatting, and session/macro recording for reproducible workflows.
68 procedures · 2 with documented limitations. Every result is computed by the open Python engine and is exportable to APA tables and reports.
| Procedure | What it does | Status | Key reference |
|---|---|---|---|
| Load Dataset | Load data from file (CSV, TSV, XLSX, SPSS .sav, Stata .dta, JSON) into active workspace. | ✅ Full | — |
| Get Dataset Info | Retrieve metadata for active dataset: row/column counts, dtypes, missing cells, columnar status. | ✅ Full | — |
| Get Data Window | Fetch windowed rows (e.g., 100 rows at offset 0) for grid rendering. | ✅ Full | — |
| Get Full Columns | Retrieve whole columns for chart building; large datasets sampled with fixed seed for row alignment. | ✅ Full | — |
| Get Variables | Inferred variable metadata (type, measure, labels, value labels) merged with engine-owned overrides. | ✅ Full | — |
| List Datasets | List all open datasets and current active dataset ID. | ✅ Full | — |
| Switch Active Dataset | Set a dataset as active (default target for all analyses and edits). | ✅ Full | — |
| Close Dataset | Remove a dataset from memory; clears associated undo/redo stacks. | ✅ Full | — |
| Duplicate Dataset | Create independent copy of dataset with all metadata and current state. | ✅ Full | — |
| Rename Dataset | Change display name (in-memory; original file unchanged). | ✅ Full | — |
| Export Dataset | Write active dataset to file (CSV, XLSX, SAV, DTA, JSON) with variable metadata. | ✅ Full | — |
| Set Cell Value | Edit single cell; coerces value to column dtype; generates undo entry. | ✅ Full | — |
| Undo Cell Edit | Reverse last cell edit (stack cap 200 per dataset). | ✅ Full | — |
| Redo Cell Edit | Restore undone cell edit. | ✅ Full | — |
| Insert Row | Insert blank row at index; structural edit clears undo stacks. | ✅ Full | — |
| Delete Row | Remove row at index; structural edit clears undo stacks. | ✅ Full | — |
| Add Variable | Append new column (all-NaN or default name auto-generated). | ✅ Full | — |
| Delete Variable | Remove column; structural edit clears undo stacks. | ✅ Full | — |
| Sort Dataset | Sort by column(s) ascending/descending; structural edit clears undo stacks. | ✅ Full | — |
| Evaluate Formula | One-off scalar formula (e.g., =A1+B1); returns value or Excel-style error code. | ✅ Full | — |
| Apply Formula to Column | Fill formula down entire column as derived variable; stored for recomputation. | ✅ Full | — |
| Recompute Derived Columns | Recalculate all derived columns after data edits in dependency order; reports #REF! for missing inputs. | ✅ Full | — |
| Get Formulas | Retrieve stored formula metadata: source, refs (letter→column), anchor row, dependencies. | ✅ Full | — |
| Set Variable Type | Convert column dtype (numeric/string/date/boolean) with honest failure accounting. | ✅ Full | — |
| Set Measurement Level | Assign measure (nominal/ordinal/scale) for SPSS Variable View parity. | ✅ Full | — |
| Set Value Labels | Map raw values to display labels (e.g., 1→'Yes', 0→'No'); empty list clears labels. | ✅ Full | — |
| Auto-Encode Categorical | SPSS-style: text categories → numeric codes with inverse map as value labels. | ✅ Full | — |
| Compute Variable | Create new variable from AST-safe expression (log, sqrt, abs, sin, cos, mean, std, +−*/). Stores result in new column. | ✅ Full | — |
| Recode Variable | Map values by pattern (e.g., '1-30'→'low', '31+'→'high') into new column. | ✅ Full | — |
| Standardize Variable | Transform to z-scores (mean 0, SD 1), min-max [0,1], or robust (median, IQR). | ✅ Full | — |
| Categorize / Bin | Convert numeric to categories: equal-width, quantile-based, or custom breaks. | ✅ Full | — |
| Filter Cases (Read-Only) | Report which rows match a condition (e.g., age > 30); read-only—use data.subset to drop. | ✅ Full | — |
| Create Dummy Variables | Expand categorical to 0/1 indicators; optionally drop first to avoid collinearity. | ✅ Full | — |
| Lag / Lead | Shift values k periods forward/backward for time-series predictors. | ✅ Full | — |
| Merge Datasets | SQL-like join (inner/left/right/outer/cross) on key column(s); reports match/unmatch counts. | ✅ Full | — |
| Reshape Long | Convert wide → long: multiple value columns become one 'value' + one 'variable' column. | ✅ Full | — |
| Reshape Wide | Convert long → wide: spread key column values across columns. | ✅ Full | — |
| Aggregate Data | Group by column(s) and compute summaries (mean/sum/count/etc.); produces new dataset. | ✅ Full | — |
| Collapse Data | Similar to aggregate; summarize grouped data (default stats: mean, SD, N). | ✅ Full | — |
| Append Datasets | Vertically stack 2+ datasets (row-bind); common columns aligned. | ✅ Full | — |
| Sort Dataset | Sort by column(s) ascending/descending; optionally replace active or create new dataset. | ✅ Full | — |
| Subset / Filter | Keep rows matching condition (AST-safe); replace active dataset. | ✅ Full | — |
| Deduplicate | Remove duplicate rows (keep first/last/none); optionally subset columns first. | ✅ Full | — |
| Transpose Dataset | Flip rows ↔ columns; optionally use one column as new row index. | ✅ Full | — |
| Split Column | Split text column by separator into multiple new columns. | ✅ Full | — |
| Concatenate Columns | Join 2+ columns into one (e.g., first + last → fullname). | ✅ Full | — |
| Rank Variable | Convert to ranks (1,2,3...); methods: average (ties split), min/max, dense, ordinal. | ✅ Full | — |
| Crosstab Counts | Build contingency table (row × col) from two categorical variables; include marginals. | ✅ Full | — |
| Pivot Longer (Complex) | Advanced wide→long: regex on column names extracts multiple groups into separate columns. | ✅ Full | — |
| Save Session | Persist all datasets, variable metadata, and output history to .stios JSON. | ✅ Full | — |
| Load Session | Restore datasets, metadata, and outputs from .stios; repopulates DatasetManager. | ✅ Full | — |
| Session Info | Read session metadata (creation date, version, dataset names, row/col counts). | ✅ Full | — |
| Session Diff | Compare two .stios files; report differences in datasets and outputs. | ✅ Full | — |
| Start Macro Recording | Begin recording user actions (transforms, analyses) for later replay. | ✅ Full | — |
| Stop Macro Recording | End current macro recording session. | ✅ Full | — |
| Save Macro | Write recorded actions to .sfmacro JSON file. | ✅ Full | — |
| Load Macro | Read .sfmacro file; return actions ready to replay. | ✅ Full | — |
| Replay Macro | Execute recorded actions in sequence (transform.compute, data.*, etc.) on current dataset. | ✅ Full | — |
| Session Undo | Reverse a logged action by ID (currently stub; per-dataset cell undo used instead). | ⚠️ Limited | — |
| Session Redo | Restore undone action by ID (currently stub; per-dataset cell redo used instead). | ⚠️ Limited | — |
| Export Script | Convert recorded macro actions to Python or R script (reproducible workflow). | ✅ Full | — |
| Regression Table | Format regression results (multiple model specs) in APA/HTML/LaTeX table. | ✅ Full | — |
| Descriptives Table | Format descriptive statistics by groups in APA/HTML/LaTeX table. | ✅ Full | — |
| Correlation Table | Format correlation matrix in APA/HTML/LaTeX; supports Pearson, Spearman, Kendall. | ✅ Full | — |
| Crosstab Table | Format contingency table (observed, expected, residuals) in APA/HTML/LaTeX. | ✅ Full | — |
| Model Comparison Table | Side-by-side regression models with fit statistics and significance stars. | ✅ Full | — |
| ANOVA Table | Format one-way/repeated-measures ANOVA table in APA/HTML/LaTeX. | ✅ Full | — |
| Factor Loading Table | Format EFA/PCA loadings with rotation (varimax, promax, etc.). | ✅ Full | — |
Notes & limitations
- Session Undo — Session-level undo not fully wired; edit.undo handles cell-level reversals.
- Session Redo — Session-level redo not fully wired; edit.redo handles cell-level reversals.