Array We There Yet?

Summary

Main finding. Array-like layouts read whole feature matrices 4.4x to 250x faster than wide layouts in 6 of 7 backends. Reading only 8 of 8,192 features: faster in 4, slower in 3 backends. The next bullets give the per-backend numbers - it is faster in Vortex (25x), Lance (14x), DuckDB (2.1x), and Parquet (1.1x); it is slower in CSV (5.7x), Zarr (3.4x), and TileDB (1.1x).

Matrix materialization loads every value into one in-memory table of numbers, ready for analysis or model training. Feature projection loads only a few chosen features, and skips the rest.

• Whole-matrix reads. Matrix materialization is faster with the array-like layout in TileDB (250x), Zarr (190x), Lance (97x), Vortex (33x), DuckDB (7.3x), and Parquet (4.4x). It is slower in CSV (1.3x). See Figure 2.
• Selecting a few features. Reading 8 of 8,192 features is faster with the array-like layout in Vortex (25x), Lance (14x), DuckDB (2.1x), and Parquet (1.1x). It is slower in CSV (5.7x), Zarr (3.4x), and TileDB (1.1x). The Zarr matrix layout is the exception worth remembering: chunked 2D arrays read whole matrices fast, but slicing a few columns out of every chunk costs more than a wide layout does. See Figure 2.
• Text packing. CSV packed arrays are slower than CSV wide for write, matrix materialization, and random rows, and for feature projection, mixed retrieval, and vector norm. Full read is the one exception, where they are faster. They are 11% larger.
• Encoding settings matter. Parquet's array layout is 29% smaller than its wide layout by default and 15% smaller with the compact profile. Default dictionary encoding inflates the wide layout. See Figure 7.
• Real-world impact. For a 1.5 GB CSV wide file, Parquet fixed_array takes 5.8 s per use instead of 26 s and costs $0.050 instead of $0.135 in egress. See Real-world example and Figure 5.
• Limits. These results come from a single machine, synthetic data, warm caches, Python bindings, and 6 repetitions per timing. See Limitations.

Figures

Real-world example

What egress is. Cloud providers charge for storing a file and, separately, for data that leaves their network. The second charge is called egress. It applies each time someone downloads a file from cloud storage to a computer outside the provider's network, and it is billed per gigabyte.

Why it matters for hosting a dataset. A team that shares a dataset pays egress each time someone downloads it. The bill grows with the number of users and with the file size, and it keeps growing after the dataset stops changing. A smaller file lowers the bill in direct proportion. Some providers offer a requester-pays option, where the person who downloads pays instead of the host.

Why it matters for using a dataset. A researcher who works on a laptop or in another cloud waits for every download, and pays the fee when the host does not. The wait starts again each time the data is read into memory. File size sets the download time. The format sets the read time.

The egress costs here are an estimate. They use one list price to show the size of the effect on a single example. Prices differ by provider, region, storage service, and volume, and they change over time.

This example asks what it costs to move and read a 1.5 GB CSV wide file, and how much the other layouts save. The file holds about 16,800 rows of 8,192 features. A use is one download followed by one read into memory. Figure 5 shows the time and egress cost.

Takeaway. With Parquet fixed_array, one use takes 5.8 s instead of 26 s and costs $0.050 instead of $0.135 in egress. Over 1,000 uses that saves 5.5 h and $85.

Size as Parquet. The same data takes 0.55 GB as a Parquet file with the array layout, 63% smaller than the 1.5 GB CSV wide file. With compact settings it takes 0.46 GB, 69% smaller. As a Parquet wide file it takes 0.81 GB, 46% smaller.

One use

Compact rows use the compact write profile, which writes smaller files and can cost time. See Write settings.

Savings over 1,000 uses

Time spent and egress spent are the totals over 1,000 uses. Time saved and egress cost saved compare a layout with CSV wide. Time is the download plus the read.

Streaming a Parquet file

Parquet stores a footer that lists where every column and row group is. A client can read the footer and then request only the byte ranges it needs from cloud storage. Egress is billed for the bytes that are sent, so a partial read costs less than a full download. A CSV file cannot be read in part by column, because every row holds every column.

The wide layout can skip features. The array layout stores all features of a row in one column, so it cannot. Row selection saves egress only when the file has several row groups and the rows you need are together. These files use one row group, so the row numbers below assume row groups of about 1,000 rows.

Reading 8 features from Parquet wide sends 4.9 MB, which costs $0.0004 in egress, against $0.135 for the CSV file. The footer is 4.1 MB (measured).

Scaling check

The benchmark data is small, so the scaling check wrote and read files of about 16,800 rows to check the scaled numbers. The tables above use the measured size and read time for these layouts.

Measured sizes are within 10% of the scaled sizes, except DuckDB duckdb_array (1.8x larger). Measured read times are within 1.5x of the scaled times, except Parquet wide (2.4x faster), Vortex fixed_array (1.6x slower), and Lance fixed_array (1.6x slower).

Egress pricing pages:

• AWS S3: the source of the $0.09 per GB used here.
• Google Cloud: egress is billed per GiB and depends on the source region.
• Azure: the first 100 GB each month is free, and the rate depends on the region.
• Cloudflare R2: no egress charges. The host pays for storage and operations instead.

Check the current page before you plan a budget.

Assumptions:

• Download speed is 100 MB/s.
• Egress costs $0.09 per GB. This is the AWS list price for data transfer out to the internet, first 10 TB each month (AWS S3 pricing). The first 100 GB each month is free on AWS. The table ignores this, so it overstates the cost at low volume.
• Sizes and read times of the layouts in the scaling check were measured on a real file of about 16,800 rows. Other rows are scaled from the benchmark data by size.
• Read times use one thread and warm caches, except for Lance and Vortex. See Limitations.
• 1 GB is 1,000,000,000 bytes.

Detailed results

The figures above show every measurement at once. The rest of this page slows down and checks the numbers behind them: the per-backend table below, then how far each layout sits from a plain NumPy file, how the results scale with row count, and how sensitive they are to compression settings. If you only load whole matrices, the summary above already has your answer. If you slice by rows or features, the next sections matter.

Key findings

Each cell compares the array-like layout with the wide layout of the same backend at 8,192 features. A negative percentage means faster or smaller. A positive percentage means slower or larger. For example, +100% means twice as slow or twice as large, and -50% means half the time or size. Feature projection reads 8 of 8,192 features, which is the best case for a wide layout. Figure 2 shows these ratios at every feature count.

These layouts used more than one thread in at least one operation: Lance fixed_array (median 0.92, up to 3.6), Lance wide (median 2, up to 3.9), Vortex fixed_array (median 2.6, up to 7.6), Vortex wide (median 1.1, up to 3.4). Parallelism is CPU time divided by wall time, so 1.0 means one busy thread. The benchmark cannot limit the native thread pools of Lance and Vortex.

Measured in CPU time, which counts every thread, matrix materialization with the array-like layout is 7x faster in Vortex (33x faster in wall time).

Access path check: for every layout, matrix materialization takes at most 3 times as long as a full read, and random rows take at most 3 times as long as matrix materialization.

Distance from a plain NumPy file

A NumPy .npy file loads into memory in 3.9 ms at 8,192 features. For matrix materialization this is a floor, because no format can be faster than a memory copy. For random rows and feature projection it is a baseline, not a floor. A row-major file is a poor layout for reading a few columns, so a value below 1x is possible. Each cell divides the time of a layout by the time of the plain file, so 1x is as fast as the plain file.

Scaling with row count

The row count grows 100x, from 2,000 to 200,000 rows, at 1,024 features. Each cell divides the time at 200,000 rows by the time at 2,000 rows. Time that grows 100x is linear in the row count. A smaller value means the time grew more slowly than the rows.

Figure 6 plots these times against the row count.

Matrix materialization grew the most in Zarr wide (390x) and the least in Lance wide (13x). Random rows grew the most in Zarr wide (190x) and the least in Lance fixed_array (1x).

CSV was measured only up to 20,000 rows.

Encoding sensitivity

Encoding and compression choices change file size and speed. This table checks whether the array-versus-wide results hold under both write profiles. Each cell shows the array-like layout relative to the wide layout of the same backend, first with the default profile and then with the compact profile. The last column says whether every measure stays on the same side of wide, better or worse, in both profiles. This does not mean that the size of the effect stays the same.

Encodings observed

The table lists the compression and encodings that each format wrote at the largest feature count. Bytes per value is the artifact size divided by the number of stored values. A raw float32 value takes 4 bytes. Most size differences between the binary formats come from default compression and encoding choices, not from the layout. Random values compress poorly, so the sizes describe this data set and not real profile data. Sizes are for 2,000 rows and can change with more rows.

Parquet turns on dictionary encoding by default. Every value in this data is unique, so the dictionary adds overhead. Parquet wide takes 5.67 bytes per value by default and 3.97 with the compact profile. DuckDB wide takes 8.26 bytes per value, about 2.1x the raw size. The benchmark did not investigate why. It stores 8,192 columns of 2,000 values each.

Appendix: storage size against time

This appendix uses synthetic random data. It shows how the compact profile moves the size and the time of each format. It is not a ranking of formats for real data.

Figure 7 plots storage size against time at the largest feature count.

Changes from the default profile to the compact profile:

The compact profile changes only the write settings in Write settings. DuckDB, Lance, and Vortex have no compact profile.

Layouts

The benchmark writes these layouts:

For packed and fixed-array layouts, the size of feature_names.json is included in the artifact size. This accounts for the feature names that a wide schema stores in its columns.

Parquet has no fixed-size list type. PyArrow writes the FixedSizeList layout as an ordinary variable-length list and restores the fixed size from the Arrow schema stored in the file metadata. Readers that do not use that schema see a variable-length list.

Storing several values in one cell is not a shortcut around the relational model. It is a documented alternative within it. Codd's First Normal Form (1970) required every value to be atomic, one number or string per cell. The NF² (non-first-normal-form) relations of Jaeschke and Schek (1982) relaxed that on purpose, allowing an attribute's value to be a list, and SQL standardized array-typed columns not long after. See References for the lineage, including the paper behind Parquet's own nested encoding.

Operations

Each run measures these operations:

Each result is stored as an absolute time. The report also stores ratios to CSV wide and ratios to the wide layout of the same backend.

Methodology

• Data. Each feature count uses a separate synthetic dataset with a fixed seed of 42. It has 2,000 rows, independent standard-normal float32 features, and three string metadata columns: sample_id, plate_id, and well_id.
• Protocol. For each backend, layout, and feature count, the benchmark writes the dataset several times to new artifacts. It then times each read operation on the last artifact. One warmup run comes before the measured runs.
• Timing. Wall time comes from time.perf_counter. CPU time comes from time.process_time, which counts every thread in the process. Checking the returned values happens after the timer stops and is not included. A read that takes less than 50 ms repeats inside one sample until the sample lasts at least 50 ms. The benchmark divides the sample time by the number of calls and stores the call count in the raw results. This removes most of the timer and thread-start noise from fast reads.
• Cache state. The benchmark does not drop the operating system page cache. All reads are warm-cache reads.
• Selections. Random rows and projected features are 128 rows and 8 features chosen with fixed seeds. The counts do not change with the feature count. Each format selects rows with its own call: take in Lance, dataset.take in Parquet, rowid in DuckDB, scan(indices=...) in Vortex, oindex in Zarr, and multi_index in TileDB. The dataset has 2,000 rows in one row group or chunk, so most formats still decode the whole block. For them random rows costs about as much as matrix materialization.
• Column requests. Wide layouts are read with a full scan followed by a column selection. The benchmark never names thousands of columns in one request. Lance takes about 1.35 ms for each column that a request names. Naming all 8,192 columns took 11 s, against 0.57 s for a full scan that returns the same values.
• Returned objects. Matrix materialization returns a NumPy array for every backend. Full read returns the native object of each library: an Arrow table, a pandas DataFrame, or a dictionary of arrays. Full read times therefore include different conversion work.
• Statistics. Each timing is the median of all repetitions. The results pool independent runs of the same code, so the error bars show the variation between runs and not only within one run. Error bars are the q25-to-q75 range. Ratios divide medians. Summary panels average ratios with the geometric mean. A measurement that varies by more than half of its median is marked with *. Pooling refuses runs from different commits or from uncommitted code.
• Threads. The benchmark asks for one thread. It enforces this for Arrow, DuckDB, TileDB, and Zarr, through the pool that each library exposes. Lance and Vortex use native thread pools that the benchmark cannot limit. The median_parallelism column in results/summary.parquet is the CPU time divided by the wall time. A value near 1.0 means one thread was busy. A larger value means more than one. For layouts that used more than one thread, the results also compare CPU time, which counts every thread.
• Correctness. Every write and read is checked for shape, float32 type, and values within 1e-6 of the source data.
• Access path check. For each layout, matrix materialization should take about as long as a full read, and random rows should not take longer than matrix materialization. The report flags a layout that takes more than three times as long. This check finds readers that ask a library for far more than an operation needs.
• Plain file. A plain NumPy .npy file is timed for the same operations. For matrix materialization it is the speed of a memory copy, so no format can be faster. For random rows and feature projection it is a baseline: a row-major file is a poor layout for reading a few columns. The report divides each layout's time by the time of the plain file.
• Row sweep. The benchmark runs again at 2,000, 20,000, and 200,000 rows and 1,024 features. CSV stops at 20,000 rows, because its files are 2.7 times the raw size and it is the slowest format. The report divides the time at the most rows by the time at the fewest, so linear growth is 100 times.
• Scaling check. The real-world example scales sizes and read times from the 2,000-row benchmark. The scaling check writes and reads a real file of about 1.5 GB for each layout of the example, and compares the measured size and read time with the scaled ones. It also measures the bytes that a client downloads to read 8 columns of a Parquet file, from the footer and the column chunk sizes.
• Encodings. Each format runs with its default settings. Formats with a verified compression setting also run a compact profile. Encoding and compression choices change file size and speed. The Encoding sensitivity table checks whether the array-versus-wide results hold under both profiles. The Encodings observed table lists what each format wrote.
• Artifact size. The size is the bytes on disk, including feature_names.json for layouts that store no column names.

Environment

Backends and access paths

The timings include each package and binding, not only the storage layout. Different language layers can change the result.

Write settings

The benchmark writes every layout with a default profile. It uses only the settings listed here. All other settings are the defaults of the package version in the Environment table.

Formats with a compression setting that the benchmark verified also run a compact profile. It writes smaller files and can cost time. The plots use only the default profile. The Encoding sensitivity table and the appendix compare the two.

Limitations

• Scale. The main benchmark has 2,000 rows, about 65 MB at the largest feature count. The row sweep goes to 200,000 rows at 1,024 features, about 820 MB, and the scaling check reads a real 1.5 GB file. All of these fit in memory. None of them shows reads that exceed memory. The sweep is one run and is not pooled. CSV was measured only up to 20,000 rows in the sweep.
• Repetitions. Each run makes 3 repetitions of each measurement. The results pool the runs listed in the Environment table. Write timings vary the most, especially TileDB writes. They can differ between runs by many times. The report marks measurements that vary by more than half of their median.
• Cache. The results do not show cold-storage or object-storage reads.
• Data. Independent random values are close to the worst case for compression. The best possible lossless size for this data is about 3.3 bytes per value. The storage sizes say little about the dictionary, run-length, and delta encodings that real profile data would use. The data has no missing values and one numeric type.
• Machine. All results come from one machine, described in the Environment table.
• Software stack. Timings include the Python bindings and the conversion to the returned object. They do not measure the storage formats alone.
• Tuning. The benchmark runs two profiles. Neither profile is tuned for the data. Other row-group sizes, chunk sizes, and codecs can change the results. Vortex, Lance, and DuckDB have no compact profile.
• Chunk size. We tested four Zarr chunk shapes separately. For the widest matrix, narrower feature chunks cut the eight-feature read from 27 ms to 11 ms, but slowed full reads and writes. They did not help for the 1,024-feature matrix. The main benchmark keeps its original chunk shape. This check used warm local files, not S3. The method and results are in spikes/001-zarr-chunk-shapes/.
• Scope. The benchmark does not measure filters, updates, concurrent access, schema changes, vector search, or tool support.

References

• Apache Parquet file format: row groups, column chunks, and the footer that lets a reader skip data.
• Apache Parquet encodings: the dictionary, run-length, and BYTE_STREAM_SPLIT encodings behind the write settings.
• Apache Arrow columnar format: the FixedSizeList type that holds the array layout in Parquet, Vortex, and Lance.
• Zarr specifications: chunked arrays and their codecs.
• TileDB documentation: dense arrays, tiles, and filters.
• DuckDB ARRAY type: the fixed-size FLOAT[N] column.
• Vortex documentation: the Vortex file format and its Python package.
• Lance documentation: the Lance file format and its Python package.
• NumPy .npy format: the file behind the NumPy floor.
• Zstandard, RFC 8878: the zstd codec in the compact profile.
• Blosc: the chunk compressor that Zarr uses in the compact profile.
• HTTP range requests, RFC 9110: how a client asks for part of a file. The pricing pages for egress are listed in Real-world example.
• Fleming, P. J. and Wallace, J. J. (1986). How not to lie with statistics: the correct way to summarize benchmark results. Communications of the ACM 29(3), 218-221. This is the case for the geometric mean of ratios.
• Codd, E. F. (1970). A Relational Model of Data for Large Shared Data Banks. Communications of the ACM 13(6). First Normal Form's atomicity rule, the rule an array-valued cell sets aside.
• Jaeschke, G., and Schek, H.-J. (1982). Remarks on the Algebra of Non First Normal Form Relations. ACM PODS. The NF² model that first formalized relation-valued attributes.
• Stonebraker, M., and Çetintemel, U. (2005). "One Size Fits All": An Idea Whose Time Has Come and Gone. ICDE. The case for specialized storage engines instead of one layout for every workload.
• Baumann, P., Misev, D., Merticariu, V., and Pham Huu, B. (2021). Array Databases: Concepts, Standards, Implementations. Journal of Big Data 8(1). The array-database tradition behind TileDB and Zarr.
• Melnik, S., et al. (2010). Dremel: Interactive Analysis of Web-Scale Datasets. PVLDB 3(1). The columnar encoding of nested fields that Parquet's array layout descends from.
• Harris, C. R., et al. (2020). Array programming with NumPy. Nature 585. The ndarray that matrix materialization reconstructs.

Terminology