79 implement the raman water vapor processing

[friedajkb]

Adressing issue #79
Already implemented:

• calibration based on model data
• water vapor mixing ratio calibration for highres and profile data

Missing features:

• adjusting the calibration constants structure to match structure of pull request 90 Refine calibration constant selection. #91
• saving highres wvmr data
• calibration constant selection currently only for regression method

[HavardStridBuholdt]

The calibration constant structure is now up to date. Additionally, a new config variable TimeInterval4LoadingDBCC is added in the latest commit to address issue #101. This variable defines the number of hours before, and after the measurement considered when loading calibration constants from the database.

@friedajkb and @martin-rdz, please have a look at my changes to see if the core physical functionality is still intact.

@ulysses78, the saving procedure for the Water Vapor retrievals is still missing. Would you be able to implement it on to this branch?

[ulysses78]

I have tryed to run this branch on rsd2....
I added to the script:

data_cube.watervaporCali()
data_cube.wvmr()

but got this error:

File "/pollyhome/Bildermacher2/PicassoPy/ppcpy/interface/picassoProc.py", line 1084, in watervaporCali
    db_table = sql_db.get_from_sql_db(
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/pollyhome/Bildermacher2/PicassoPy/ppcpy/io/sql_interaction.py", line 120, in get_from_sql_db
    for index, row in df[df.cali_method == 'Model_Profile_Method'].iterrows():
                         ^^^^^^^^^^^^^^
  File "/pollyhome/Bildermacher2/picassopy_venv/lib64/python3.11/site-packages/pandas/core/generic.py", line 6206, in __getattr__
    return object.__getattribute__(self, name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DataFrame' object has no attribute 'cali_method'. Did you mean: '_arith_method'?

[HavardStridBuholdt]

@ulysses78 the issue is that the db-file it is trying to load from already has a table called wv_calibration_constants but the column names and structure differ compared to what PicassoPy assumes.

Currently the structure of the wv_calibration_constant table in PicassoPy looks like this:

• id : integer
• cali_start_time : text
• cali_stop_time : text
• wv_const : real
• uncertainty_wv_const : real
• used_for_processing : integer -- 1 or 0
• wavelength : text -- '407 nm'
• nc_zip_file : text
• polly_type : text
• cali_method : text -- 'Model_Profile_Method' or 'Model_Regression_Method' etc.
• telescope : text -- 'FR'

While the structure of the db-file on rsd2 and whats used in Picasso (the Matlab version) looks like this:

• id : integer
• cali_start_time : text
• cali_stop_time : text
• standard_instrument : text -- 'RPG-Radiometer Physics HATPRO' etc.
• standard_instrument_meas_time : text
• wv_const : real
• uncertainty_wv_const : real
• nc_zip_file : text
• polly_type : text

The crash could be fixed by checking if the column we are trying to access exists before accessing it. However, I believe we need to discuss which structure to use. Do we want to keep the current structure (which is basically just of copy of the lidar_calibration_constant table structure) or do we want to change to a structure similar to what is used in Matlab? If we choose to use the Matlab structure, we should add used_for_processing column. Thus, the table will anyways need to be remade.

Which structure would you prefer @friedajkb, @martin-rdz, and @HolgerPollyNet.

[friedajkb]

@HavardStridBuholdt @ulysses78 I'd prefer to keep the current structure used in PicassoPy. Then it is more consistent throughout all calibrations done in PicassoPy. And we also use a slightly different approach than in the Matlab version, for which I think the current structure makes more sense.

[martin-rdz]

Totally agree with frieda.
The databases are not backward compatible in general, because they are also using the unique column statement, to make values overwritable without too much effort (for example)