Commit abbe36d
committed
ALP: Add pipeline integration, reader buffer reuse, and validation
Writer pipeline integration:
- Use CapacityByteArrayOutputStream for encoded vector storage instead
of List<byte[]>, integrating with Parquet's memory management
- Use BytesInput.concat() for zero-copy page assembly
- Accept (initialCapacity, pageSize, ByteBufferAllocator) constructor
params; factory now passes pipeline properties
Reader memory efficiency:
- Allocate decoded buffer once in initFromPage() and reuse across all
vector decodes, eliminating per-vector float[]/double[] allocations
- Improves decode throughput 5-24% across all datasets
Reader validation:
- Validate logVectorSize bounds (MIN_LOG to MAX_LOG)
- Validate non-negative element count
- Validate skip(n) bounds1 parent 385d053 commit abbe36d
5 files changed
Lines changed: 147 additions & 95 deletions
File tree
- parquet-column/src/main/java/org/apache/parquet/column/values
- alp
- factory
Lines changed: 22 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
36 | 40 | | |
37 | 41 | | |
38 | 42 | | |
| |||
42 | 46 | | |
43 | 47 | | |
44 | 48 | | |
45 | | - | |
| 49 | + | |
46 | 50 | | |
47 | 51 | | |
48 | 52 | | |
| |||
67 | 71 | | |
68 | 72 | | |
69 | 73 | | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
70 | 81 | | |
71 | 82 | | |
72 | 83 | | |
| |||
76 | 87 | | |
77 | 88 | | |
78 | 89 | | |
| 90 | + | |
79 | 91 | | |
80 | 92 | | |
81 | 93 | | |
| |||
90 | 102 | | |
91 | 103 | | |
92 | 104 | | |
| 105 | + | |
| 106 | + | |
93 | 107 | | |
94 | 108 | | |
95 | 109 | | |
| |||
99 | 113 | | |
100 | 114 | | |
101 | 115 | | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
102 | 120 | | |
103 | 121 | | |
104 | 122 | | |
| |||
110 | 128 | | |
111 | 129 | | |
112 | 130 | | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
113 | 134 | | |
Lines changed: 7 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
| 26 | + | |
25 | 27 | | |
26 | 28 | | |
27 | 29 | | |
28 | 30 | | |
29 | 31 | | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
30 | 37 | | |
31 | 38 | | |
32 | 39 | | |
| |||
47 | 54 | | |
48 | 55 | | |
49 | 56 | | |
50 | | - | |
51 | 57 | | |
52 | 58 | | |
53 | 59 | | |
| |||
Lines changed: 7 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
| 26 | + | |
25 | 27 | | |
26 | 28 | | |
27 | 29 | | |
28 | 30 | | |
29 | 31 | | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
30 | 37 | | |
31 | 38 | | |
32 | 39 | | |
| |||
47 | 54 | | |
48 | 55 | | |
49 | 56 | | |
50 | | - | |
51 | 57 | | |
52 | 58 | | |
53 | 59 | | |
| |||
0 commit comments