-
Notifications
You must be signed in to change notification settings - Fork 32
feat: add decoding as a multi-threaded CLARA engine #1229
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
baltzell
wants to merge
45
commits into
development
Choose a base branch
from
decoder-engine
base: development
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
45 commits
Select commit
Hold shift + click to select a range
88a37c8
add convenience method
baltzell 81a942f
remove unused stuff
baltzell 5b04c7f
add decoding engine
baltzell e007bff
add it to the clara yaml
baltzell 4b4f4ed
use a pool
baltzell 7533d1b
hmm
baltzell d2effb6
allow decoder instances to share ConstantsManagers
baltzell 813df72
share ConstantsManagers
baltzell b67e68e
kludge test
baltzell a27e855
cleanup
baltzell b11aae4
try this
baltzell 1ac9d03
inherit ConstantsManagers
baltzell 3f60c48
cleanup
baltzell d788f4a
only check tables if not shared
baltzell 7ce4da4
higher ports on macos, cleanup process dpe process
baltzell 5274c3d
remove ineffective pid trap
baltzell ba52c75
Revert "higher ports on macos, cleanup process dpe process"
baltzell 269ff08
Revert "fix job name"
baltzell 030152d
Revert "decouple ubuntu/macos builds to reduce wait"
baltzell 2e3fefe
Revert "remove unnecessary reader class"
baltzell 676df4d
Revert "remove example engine"
baltzell 8986cf4
restore reported data type
baltzell 5a693e3
rename class
baltzell 756087d
undo
baltzell 4036a67
fix rebase oops
baltzell ac20ac1
rename class
baltzell d944fa4
fix rebase oops
baltzell 5a1ff60
cleanup
baltzell 7b33a15
cleanup
baltzell 313b4be
stf
baltzell 4b3b4e6
just events, not per thread
baltzell 84d0662
reduce constants sharing
baltzell da0aa0d
bugfix
baltzell 159eb5e
share one RCDBManager
baltzell 0ed9fa6
bugfix
baltzell 8c6cb24
restore
baltzell 2c95583
DecoderEngine: just pass along HIPO events
baltzell 24bc494
add Clas12Reader, no decoding
baltzell fa2facf
dummy initialization value
baltzell b9ca5aa
bugfix
baltzell b768d23
split I/O service into separate PR
baltzell c29d9ba
split pull requests
baltzell 40630b0
cleanup, avoid class variable
baltzell 71d85f6
remove debugging leftover
baltzell b5d51dc
revert to hard-coded byte order
baltzell File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
113 changes: 113 additions & 0 deletions
113
common-tools/clas-reco/src/main/java/org/jlab/clas/reco/DecoderEngine.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| package org.jlab.clas.reco; | ||
|
|
||
| import java.util.Set; | ||
| import java.util.HashSet; | ||
| import java.nio.ByteBuffer; | ||
| import java.nio.ByteOrder; | ||
| import java.util.concurrent.ArrayBlockingQueue; | ||
| import java.util.concurrent.BlockingQueue; | ||
| import org.jlab.clara.base.ClaraUtil; | ||
| import org.jlab.clara.engine.Engine; | ||
| import org.jlab.clara.engine.EngineData; | ||
| import org.jlab.clara.engine.EngineDataType; | ||
| import org.jlab.clara.engine.EngineStatus; | ||
| import org.jlab.detector.decode.CLASDecoder; | ||
| import org.jlab.io.evio.EvioDataEvent; | ||
| import org.jlab.io.hipo.HipoDataEvent; | ||
| import org.jlab.jnp.hipo4.data.SchemaFactory; | ||
| import org.json.JSONObject; | ||
|
|
||
| /** | ||
| * | ||
| * @author baltzell | ||
| */ | ||
| public class DecoderEngine implements Engine { | ||
|
|
||
| static final int POOL_SIZE = 64; | ||
| static final Set<EngineDataType> ED_TYPES = ClaraUtil.buildDataTypes( | ||
| Clas12Types.EVIO,Clas12Types.HIPO,EngineDataType.JSON,EngineDataType.STRING); | ||
|
|
||
| SchemaFactory schema; | ||
| BlockingQueue<CLASDecoder> pool; | ||
| int constantsShared = 16; | ||
|
|
||
| public DecoderEngine() { | ||
| schema = new SchemaFactory(); | ||
| schema.initFromDirectory(System.getenv("CLAS12DIR") + "/etc/bankdefs/hipo4"); | ||
| } | ||
|
|
||
| @Override | ||
| public Set<EngineDataType> getInputDataTypes() { return ED_TYPES; } | ||
| @Override | ||
| public Set<EngineDataType> getOutputDataTypes() { return ED_TYPES; } | ||
| @Override | ||
| public EngineData executeGroup(Set<EngineData> set) { return null; } | ||
| @Override | ||
| public Set<String> getStates() { return new HashSet<>(); } | ||
| @Override | ||
| public String getDescription() { return "decoder engine"; } | ||
| @Override | ||
| public String getVersion() { return "1.0"; } | ||
| @Override | ||
| public String getAuthor() { return "baltzell"; } | ||
| @Override | ||
| public void reset() {} | ||
| @Override | ||
| public void destroy() {} | ||
|
|
||
| @Override | ||
| public EngineData configure(EngineData ed) { | ||
| JSONObject json = new JSONObject(ed.getData()); | ||
| pool = new ArrayBlockingQueue<>(POOL_SIZE); | ||
| CLASDecoder d0 = null; | ||
| for (int i=0; i<POOL_SIZE; i++) { | ||
| CLASDecoder d; | ||
| if (i % constantsShared == 0) { | ||
| d0 = new CLASDecoder(); | ||
| if (json.has("variation")) d0.setVariation(json.getString("variation")); | ||
| if (json.has("timestamp")) d0.setVariation(json.getString("timestamp")); | ||
| d = d0; | ||
| } | ||
| else { | ||
| d = new CLASDecoder(d0); | ||
| } | ||
| pool.add(d); | ||
| } | ||
| return ed; | ||
| } | ||
|
|
||
| @Override | ||
| public EngineData execute(EngineData input) { | ||
|
|
||
| EngineData output = input; | ||
|
|
||
| // if it's EVIO, decode it, otherwise just pass it along | ||
| if (input.getMimeType().equals("binary/data-evio")) { | ||
| EvioDataEvent evio; | ||
| try { | ||
| ByteBuffer bb = (ByteBuffer) input.getData(); | ||
| //evio = new EvioDataEvent(bb.array(), bb.order()); | ||
| evio = new EvioDataEvent(bb.array(), ByteOrder.LITTLE_ENDIAN); | ||
| } catch (Exception e) { | ||
| String msg = String.format("Error reading input event%n%n%s", ClaraUtil.reportException(e)); | ||
| output.setStatus(EngineStatus.ERROR); | ||
| output.setDescription(msg); | ||
| return output; | ||
| } | ||
| HipoDataEvent hipo; | ||
| try { | ||
| CLASDecoder d = pool.take(); | ||
| hipo = new HipoDataEvent(d.getDecodedEvent(evio),schema); | ||
| pool.put(d); | ||
| output.setData("binary/data-hipo", hipo.getHipoEvent()); | ||
| } catch (Exception e) { | ||
| String msg = String.format("Error processing input event%n%n%s", ClaraUtil.reportException(e)); | ||
| output.setStatus(EngineStatus.ERROR); | ||
| output.setDescription(msg); | ||
| return output; | ||
| } | ||
| } | ||
|
|
||
| return output; | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.