"WARNING:  out of shared memory" when running test:vmdb:setup with PARALLEL and PG 9.6

Trying to run PARALLEL=1 bundle exec bin/rake test:vmdb:setup (with no existing vmdb_test* databases), I get..

LINE 1: INSERT INTO "schema_migrations" ("version") VALUES ($1) RETU...
LINE 6:                            strpos(split_part(pg_get_expr(def...

All 8 databases get created. But apparently the ar_internal_metadata table is sometimes empty, and when that happens, you get a ActiveRecord::NoEnvironmentInSchemaError running test:vmdb:setup and that database gets skipped. Dropping it manually fixes that though.
@jrafanie
I'm using PostgreSQL 9.6.1, possibly related?

@himdel Definitely related. Still looking in to this to see if it's a fixable issue programmatically but I recommend upping max_locks_per_transaction to 128 (default is 64) in your postgresql.conf (that's in /usr/local/var/postgres if you're on MacOS), which should fix the issue.

@chrisarcand aah, thanks! Upping max_locks_per_transaction works for me too :) (and that's /etc/postgresql/9.6/main/postgresql.conf - Debian ;)).
Any idea why the difference? The default has always been 64 it seems..

Wow, thanks for looking into this @chrisarcand @himdel, very interesting

So.. do we want to track 9.6 problems?  Apart from this one, I also have 2 failing specs..
rspec ./spec/models/vmdb_database_connection_spec.rb:133 # VmdbDatabaseConnection.log_statistics normal
got: "MIQ(VmdbDatabaseConnection.log_statistics) Unable to log stats, 'undefined local variable or method `waiting' for #<VmdbDatabaseConnection:0x0055798d7efb10>\n"

rspec ./spec/models/vmdb_database_spec.rb:75 # VmdbDatabase.report_client_connections will return an array of hashes and verify hash keys for client connections query
   ActiveRecord::StatementInvalid:
       PG::UndefinedColumn: ERROR:  column "waiting" does not exist
       LINE 4:                        , waiting       AS is_waiting

Any idea why the difference? The default has always been 64 it seems..

Yeah that's the part I don't understand yet 😕  The only shot in the dark idea I have is that with the new parallel queries feature, some internal queries add to the number of table writes at the same time and finally push it over the edge of the current 64 setting.

I can create an issue.. :)

@himdel Yeah go ahead and make a separate one, let's leave this issue specific to this problem.

This one feels like a local config issue and probably doesn't have a solution within our application.
I'm not seeing this in PG 10. How do you want to handle this @himdel ? I'd be fine closing it ...

@carbonin You're right .. for 9.6, I had to up the value in the config, for 11, I'm running the default config again.
So, I'm fine with closing, except: our default config for 9.5 does not override max_locks_per_trasnaction, and 64 is the default value.
So if we ever build an appliance with 9.6, the config change would still be needed, right?
But if the plan is to skip ahead to 10 or 11 (or if the problem can't happen on our appliances for some reason), I'm fine with closing too :)

Based on the PG source it looks like the default was last changed 14 years ago (https://github.com/postgres/postgres/blame/REL_10_STABLE/src/backend/utils/misc/postgresql.conf.sample#L601). So I bet it's an issue with whatever distribution was being installed rather than the PG version.
I'll go ahead and close this.


documentation: how to configure SemaphoreCI

It should be documented how to configure SemaphoreCI (platform, commands).
See an example of such documentation:
https://github.com/coreos/rkt/blob/cf0aa2e4c6062e76c6a18a8af99641e84aac9ecc/tests/README.md#build-settings
/cc @schu @2opremio

Thanks (I could have used this)

We now explain that in the stage1-builder README: https://github.com/kinvolk/stage1-builder#using-custom-built-stage1-kvm-images-on-semaphore-ci


Refactor Refresh Emissions Views Database Procedures

The emissions view refresh procedures ported from the MS SQL client and used in the workspace do not perform well when used in the official schema due to the vast amount of data. Need to refactor the refresh procedures to be more performant overall.
There are 3 functions that pivot the derived_hrly_value, monitor_hrly_value, & hrly_param_fuel_flow tables so that the values needed for each parameter are a single row for each hour reported.

get_derived_hourly_values_pivoted
get_monitor_hourly_values_pivoted
get_hourly_param_fuel_flow_pivoted

NOTE: The above functions are dynamic functions that return a dynamic list of columns based on the parameter codes passed into the function from the caller. If you add a column to be returned then anything that uses it must be updated to expected that column.
Instead of joining multiple times on derived_hrly_value, monitor_hrly_value, and/or hrly_param_fuel_flow which has 100+ million records based on different parameters. Replace all these joins with one join to the appropriate pivot functions above passing in the parameters that data is needed for.
The following have been refactored to use the pivot approach and can be used as examples...

[x] refresh_emission_view_all
[x] refresh_emission_view_co2appd
[x] refresh_emission_view_co2calc
[x] refresh_emission_view_co2cems
[x] refresh_emission_view_hiappd

The following had previously been refactored to use an approach that does not always work well for every case and needs to be refactored to use the pivot approach...

[ ] refresh_emission_view_hicems
[ ] refresh_emission_view_hiunitstack
[ ] refresh_emission_view_lme
[ ] refresh_emission_view_massoilcalc
[ ] refresh_emission_view_matshcl
[ ] refresh_emission_view_matshf
[ ] refresh_emission_view_matshg
[ ] refresh_emission_view_matsso2
[ ] refresh_emission_view_moisture
[ ] refresh_emission_view_noxappemixedfuel
[ ] refresh_emission_view_noxappesinglefuel

The following are the original ported procedures that have not been refactored at all and needs to be refactored to use the pivot approach...

[ ] refresh_emission_view_noxmasscems
[ ] refresh_emission_view_noxratecems
[ ] refresh_emission_view_so2appd
[ ] refresh_emission_view_so2cems

The following cannot use the pivot approach but need to be looked at from a performance perspective...

[ ] refresh_emission_view_matssorbent
[ ] refresh_emission_view_matsweekly
[ ] refresh_emission_view_co2dailyfuel
[ ] refresh_emission_view_dailycal
[ ] refresh_emission_view_otherdaily

NOTE: Refactor the camdecmps schema (official) procedures and test in the TEST or CDC database which has all of the production data and then copy them to the camdecmpswks schema (workspace) replacing the referenced schema from camdecmps to camdecmpswks
The following procedures are not persisted views and instead uses an actual view for the data so these refresh procedures are just refreshing the counts and should not need refactoring...

refresh_emission_view_count
refresh_emission_view_dailybackstop
refresh_emission_view_ltff
refresh_emission_view_nsps4t

CVP will not work on this ticket. DPC will complete reamaining work

Jason, is it possible to load the Emission View data from Workspace into Official when emission reports are submitted instead of reproducing the values?  This approach would need to account for the following:

Not allowing the failure to load Emission View data to prevent a successful submission.
Still providing the ability to refresh the Official Emission View data for Official data.
The second issue would eventually require improving the efficiency of the Official process, but the need might not be as urgent.

ECMPS Priorities Follow-up Emission View Generation FYI.txt

[ ] refresh_emission_view_noxmasscems
[ ]  refresh_emission_view_noxratecems
[ ] refresh_emission_view_so2appd
[ ] refresh_emission_view_so2cems

PR: https://github.com/US-EPA-CAMD/easey-db-scripts/pull/10

need to discuss with @ergjustin

Related Tickets:

https://github.com/US-EPA-CAMD/easey-ui/issues/5960

Potential Alternate Solution for
As far as I can tell the pivot function method would not use indexes to join the DHV and MHV data.  This might not cause an execution hit, but worries me.  The following method achieves a similar result as the pivot function method, but should join using indexes.
View Emissions SO2 CEMS Query
select  cor.Oris_Code,
        cor.Facility_Name,
        cor.Locations,
        /* Real Columns */
        cor.Mon_Plan_Id,
        cor.Mon_Loc_Id,
        cor.Rpt_Period_Id,
        cor.Begin_Date as Date,
        cor.Begin_Hour as Hour,
        cor.Op_Time,
        cor.Hr_Load as Unit_Load,
        cor.Load_Uom_Cd as Load_Uom,
        cor.So2c_Mhv_Unadjusted_Hrly_Value as Unadj_So2,
        cor.So2c_Mhv_Applicable_Bias_Adj_Factor as So2_Baf,
        cor.So2c_Mhv_Adjusted_Hrly_Value as Rpt_Adj_So2,
        cor.So2c_Mhv_Calc_Adjusted_Hrly_Value as Calc_Adj_So2,
        cor.So2c_Mhv_Modc_Cd as So2_Modc,
        cor.So2c_Mhv_Pct_Available as So2_Pma,
        cor.Flow_Mhv_Unadjusted_Hrly_Value as Unadj_Flow,
        cor.Flow_Mhv_Applicable_Bias_Adj_Factor as Flow_Baf,
        cor.Flow_Mhv_Adjusted_Hrly_Value as Rpt_Adj_Flow,
        cor.Flow_Mhv_Calc_Adjusted_Hrly_Value as Adj_Flow_Used,
        cor.Flow_Mhv_Modc_Cd as Flow_Modc,
        cor.Flow_Mhv_Pct_Available as Flow_Pma,
        cor.So2_Dhv_Calc_Pct_Moisture as Pct_Wrong_Used,
        case
            when cor.So2_Dhv_Calc_Pct_Moisture is null then null
            when cor.h2o_dhv_Modc_Cd is not null then cor.h2o_dhv_Modc_Cd
            when cor.h2o_mhv_Modc_Cd is not null then cor.h2o_mhv_Modc_Cd
            else 'DF'
        end as Source_H2o_Value,
        frm.Equation_Cd as So2_Formula_Code,
        cor.So2_Dhv_Adjusted_Hrly_Value as Rpt_So2_Mass_Rate,
        cor.So2_Dhv_Calc_Adjusted_Hrly_Value as Calc_So2_Mass_Rate,
        cor.Hi_Dhv_Calc_Adjusted_Hrly_Value,
        cor.H2o_Dhv_Modc_Cd,
        cor.So2r_Dhv_Adjusted_Hrly_Value,
        cor.H2o_Mhv_Modc_Cd,
        case (frm.Equation_Cd)
            when 'F-23' then cor.Hi_Dhv_Calc_Adjusted_Hrly_Value
        end as Calc_Hi_Rate,
        case (frm.Equation_Cd)
            when 'F-23' then
                coalesce
                (
                    cor.So2r_Dhv_Adjusted_Hrly_Value,
                    (
                        select  max( Default_Value )
                          from  camdecmps.MONITOR_DEFAULT def
                         where  def.Mon_Loc_Id = cor.Mon_Loc_Id
                           and  def.Parameter_Cd = 'SO2R'
                           and  def.Default_Purpose_Cd = 'F23'
                           and  ( def.Begin_Date  + interval '1' HOUR * def.Begin_Hour ) <= ( cor.Begin_Date  + interval '1' HOUR * cor.Begin_Hour )
                           and  ( ( def.End_Date is null) or ( def.End_Date  + interval '1' HOUR * def.End_Hour ) <= ( cor.Begin_Date  + interval '1' HOUR * cor.Begin_Hour ) )
                    )
                )
        end as Default_So2_Emission_Rate
  from  (
            select  sel.Oris_Code,
                    sel.Facility_Name,
                    sel.Locations,
                    sel.Mon_Plan_Id,
                    sel.Rpt_Period_Id,
                    sel.Mon_Loc_Id,
                    sel.Begin_Date,
                    sel.Begin_Hour,
                    sel.Op_Time,
                    sel.Hr_Load,
                    sel.Load_Uom_Cd,
                    max( case when dhv.Parameter_Cd = 'SO2'  then dhv.Hour_Id end ) as So2_Dhv_Hour_Id,
                    max( case when dhv.Parameter_Cd = 'SO2'  then dhv.Calc_Pct_Moisture end ) as So2_Dhv_Calc_Pct_Moisture,
                    max( case when dhv.Parameter_Cd = 'SO2'  then dhv.Adjusted_Hrly_Value end ) as So2_Dhv_Adjusted_Hrly_Value,
                    max( case when dhv.Parameter_Cd = 'SO2'  then dhv.Calc_Adjusted_Hrly_Value end ) as So2_Dhv_Calc_Adjusted_Hrly_Value,
                    max( case when dhv.Parameter_Cd = 'SO2'  then dhv.Mon_Form_Id end ) as So2_Dhv_Mon_Form_Id,
                    max( case when dhv.Parameter_Cd = 'HI'   then dhv.Calc_Adjusted_Hrly_Value end ) as Hi_Dhv_Calc_Adjusted_Hrly_Value,
                    max( case when dhv.Parameter_Cd = 'H2O'  then dhv.Modc_Cd end ) as H2o_Dhv_Modc_Cd,
                    max( case when dhv.Parameter_Cd = 'H2O'  then dhv.Pct_Available end ) as H2o_Dhv_Pct_Available,
                    max( case when dhv.Parameter_Cd = 'SO2R' then dhv.Adjusted_Hrly_Value end ) as So2r_Dhv_Adjusted_Hrly_Value,
                    max( case when mhv.Parameter_Cd = 'FLOW' then mhv.Hour_Id end ) as Flow_Mhv_Hour_Id,
                    max( case when mhv.Parameter_Cd = 'FLOW' then mhv.Unadjusted_Hrly_Value end ) as Flow_Mhv_Unadjusted_Hrly_Value,
                    max( case when mhv.Parameter_Cd = 'FLOW' then mhv.Applicable_Bias_Adj_Factor end ) as Flow_Mhv_Applicable_Bias_Adj_Factor,
                    max( case when mhv.Parameter_Cd = 'FLOW' then mhv.Adjusted_Hrly_Value end ) as Flow_Mhv_Adjusted_Hrly_Value,
                    max( case when mhv.Parameter_Cd = 'FLOW' then mhv.Calc_Adjusted_Hrly_Value end ) as Flow_Mhv_Calc_Adjusted_Hrly_Value,
                    max( case when mhv.Parameter_Cd = 'FLOW' then mhv.Modc_Cd end ) as Flow_Mhv_Modc_Cd,
                    max( case when mhv.Parameter_Cd = 'FLOW' then mhv.Pct_Available end ) as Flow_Mhv_Pct_Available,
                    max( case when mhv.Parameter_Cd = 'SO2C' then mhv.Unadjusted_Hrly_Value end ) as So2c_Mhv_Unadjusted_Hrly_Value,
                    max( case when mhv.Parameter_Cd = 'SO2C' then mhv.Applicable_Bias_Adj_Factor end ) as So2c_Mhv_Applicable_Bias_Adj_Factor,
                    max( case when mhv.Parameter_Cd = 'SO2C' then mhv.Adjusted_Hrly_Value end ) as So2c_Mhv_Adjusted_Hrly_Value,
                    max( case when mhv.Parameter_Cd = 'SO2C' then mhv.Calc_Adjusted_Hrly_Value end ) as So2c_Mhv_Calc_Adjusted_Hrly_Value,
                    max( case when mhv.Parameter_Cd = 'SO2C' then mhv.Modc_Cd end ) as So2c_Mhv_Modc_Cd,
                    max( case when mhv.Parameter_Cd = 'SO2C' then mhv.Pct_Available end ) as So2c_Mhv_Pct_Available,
                    max( case when mhv.Parameter_Cd = 'H2O'  then mhv.Modc_Cd end ) as H2o_Mhv_Modc_Cd
              from  (
                        select  fac.Oris_Code,
                                fac.Facility_Name,
                                (
                                    select  string_agg( coalesce( unt.Unitid, stp.Stack_Name ), ', ' order by case when unt.Unit_Id is null then 0 else 1 end, coalesce( unt.Unitid, stp.Stack_Name ) )
                                      from  camdecmps.MONITOR_PLAN_LOCATION mpl
                                            join camdecmps.MONITOR_LOCATION loc on loc.Mon_Loc_Id = mpl.Mon_Loc_Id
                                            left join camd.UNIT unt on unt.Unit_Id = loc.Unit_Id
                                            left join camdecmps.STACK_PIPE stp on stp.Stack_Pipe_Id = loc.Stack_Pipe_Id
                                     where  mpl.Mon_Plan_Id = pln.Mon_Plan_Id
                                ) as Locations,
                                hod.Begin_Date,
                                hod.Begin_Hour,
                                hod.Op_Time,
                                hod.Hr_Load,
                                hod.Load_Uom_Cd,
                                ems.Mon_Plan_Id,
                                ems.Rpt_Period_Id,
                                mpl.Mon_Loc_Id,
                                hod.Hour_Id
                          from  camdecmps.EMISSION_EVALUATION ems
                                join camdecmps.MONITOR_PLAN pln on pln.Mon_Plan_Id = ems.Mon_Plan_Id
                                join camd.PLANT fac on fac.Fac_Id = pln.Fac_Id
                                join camdecmps.MONITOR_PLAN_LOCATION mpl on mpl.Mon_Plan_Id = ems.Mon_Plan_Id
                                join camdecmps.HRLY_OP_DATA hod on hod.Mon_Loc_Id = mpl.Mon_Loc_Id and hod.Rpt_Period_Id = ems.Rpt_Period_Id
                         where  ems.Mon_Plan_Id = 'TWCORNEL5-488E42008B434177BC7D7BFF138D18EF'
                           and  ems.Rpt_Period_Id = 109
                           and  pln.End_Rpt_Period_Id is null
                    ) sel
                    join camdecmps.DERIVED_HRLY_VALUE dhv on dhv.Hour_Id = sel.Hour_Id and dhv.Parameter_Cd in ( 'SO2', 'HI', 'H2O', 'SO2R' )
                    join camdecmps.MONITOR_HRLY_VALUE mhv on mhv.Hour_Id = sel.Hour_Id and mhv.Parameter_Cd in ( 'FLOW', 'SO2C', 'H20' )
             group
                by  sel.Oris_Code,
                    sel.Facility_Name,
                    sel.Locations,
                    sel.Mon_Plan_Id,
                    sel.Rpt_Period_Id,
                    sel.Mon_Loc_Id,
                    sel.Begin_Date,
                    sel.Begin_Hour,
                    sel.Op_Time,
                    sel.Hr_Load,
                    sel.Load_Uom_Cd
        ) cor
        left join camdecmps.MONITOR_FORMULA frm on frm.Mon_Form_Id = cor.So2_Dhv_Mon_Form_Id
 where  cor.So2_Dhv_Hour_Id is not null
   and  cor.Flow_Mhv_Hour_Id is not null

All involved emissions tables have been analyzed on dev and the refresh procedures are running much faster. Code has been checked for all updated procedures.

@djw4erg to provide scripts which can be used to first clear data from affected views prior to testing.

For stack/pipe locations, the Hourly Heat Input CEMS and Hourly CO2 CEMS Views have counts which appear to be the sum for all locations in the monitoring plan even if HI and CO2 is not measured at all locations. Ex. ORIS 1356, CS023, 2, 3. The counts equal 6624 and should be 2208. HI and CO2 is only measured at CS023. These views are also showing data for units 2 and 3 and should not. This also affects other locations that do not have HI or CO2 CEMS (ex. LME, Appendix D with only NOXR CEMS). This also affects both the public and workspace views.
MATS HG View

HG Formula Code column is always null (both public and workspace views; ex. ORIS 1356, CS023).
Rptd. % Diluent, Diluent Parameter, Diluent MODC, and F-Factor columns null (both public and workspace views; ex. ORIS 564, 1).

Hourly NOx Mass CEMS View does not display any data (both public and workspace views; ex. ORIS 880041, X015).
%Diluent Used in Calc. and Calc. HCl Rate columns missing data in MATS HCL View in the workspace (Calc. HCl Rate column in particular should be populated before and after an evaluation; ex. ORIS 4125, 9).

Handling Testing Results

Hourly Heat Input CEMS and Hourly CO2 CEMS Views
MATS HG View
Hourly NOx Mass CEMS View
MATS HCL View
REFRESH_EMISSIONS_VIEWS Execution Stuck and Missing Column Data

Handling Testing Results Suggested Hourly NOx Mass CEMS View with Reduced Execution Time
select  hod.mon_plan_id,
        hod.mon_loc_id,
        hod.rpt_period_id,
        camdecmps.format_date_hour( hod.begin_date, hod.begin_hour, null ) AS date_hour,
        hod.op_time,
        hod.hr_load as unit_load,
        hod.load_uom_cd as load_uom,
        noxc.Unadjusted_Hrly_Value as unadjj_nox,
        noxc.Applicable_bias_adj_factor as calc_nox_baf,
        noxc.adjusted_hrly_value as rpt_adj_nox,
        noxc.calc_adjusted_hrly_value as rpt_adj_nox,
        noxc.modc_cd as nox_modc,
        noxc.pct_available as nox_p0ma,
        flow.unadjusted_hrly_value as unadj_flow,
        flow.applicable_bias_adj_factor as calc_flow_baf,
        flow.adjusted_hrly_value as rpt_adj_flow,
        flow.calc_adjusted_hrly_value as adj_flow_used,
        flow.modc_cd as flow_modc,
        flow.pct_available as flow_pma,
        nox.calc_pct_moisture as pct_h2o_used,
        case
            when ( noxr.calc_pct_moisture is not null ) then
                case
                    when ( h2o_m.modc_cd is not null ) then h2o_m.modc_cd
                    when ( h2o_d.modc_cd is not null ) then h2o_d.modc_cd
                    else 'DF'
                end
            else null
        end as source_h2o_value,
        frm.equation_cd as noxmass_formula_code,
        nox.adjusted_hrly_value as rpt_nox_mass,
        nox.calc_adjusted_hrly_value as calc_nox_Mass
  from  (
            select  hod.hour_id,
                    mpl.MON_PLAN_ID,
                    hod.MON_LOC_ID,
                    hod.RPT_PERIOD_ID,
                    hod.BEGIN_DATE,
                    hod.BEGIN_HOUR,
                    hod.OP_TIME,
                    hod.HR_LOAD,
                    hod.LOAD_UOM_CD
              from  camdecmps.MONITOR_PLAN_LOCATION mpl
                    join camdecmps.HRLY_OP_DATA hod on hod.mon_loc_id = mpl.mon_loc_id
             where  mpl.mon_plan_id = 'MDC-EC0E1E1664394A6DB9DF0DBCFD5C93C1'
               and  hod.rpt_period_id = 123
         ) AS hod
         join camdecmps.DERIVED_HRLY_VALUE nox
           on nox.hour_id = hod.hour_id
          and nox.parameter_cd = 'NOX'
         join camdecmps.MONITOR_HRLY_VALUE flow
           on flow.hour_id = hod.hour_id
          and flow.parameter_cd = 'FLOW'
         join camdecmps.MONITOR_FORMULA frm
           on frm.mon_form_id = nox.mon_form_id
          and frm.equation_cd like 'F-26%'
         left join camdecmps.MONITOR_HRLY_VALUE noxc
           on noxc.hour_id = hod.hour_id
          and noxc.parameter_cd = 'NOXC'
         left join camdecmps.MONITOR_HRLY_VALUE h2o_m
           on h2o_m.hour_id = hod.hour_id
          and h2o_m.parameter_cd = 'H2O'
         left join camdecmps.DERIVED_HRLY_VALUE h2o_d
           on h2o_d.hour_id = hod.hour_id
          and h2o_d.parameter_cd = 'H2O'
         left join camdecmps.DERIVED_HRLY_VALUE noxr
           on noxr.hour_id = hod.hour_id
          and noxr.parameter_cd = 'NOXR'

Related to #6203.

Closing Comment
Additional changes and testing will occur a s part of #6410.


Scan finished but the json file isnt created

Tried changing the folder it would save to, still doesn't appear, but from what I can tell, it finished scanning. It just closes itself afterwards, and I can't find the json file in the folder it's supposed to be in.

I believe this happens when a character is not found. Please let me know if this still occurs in newer versions.

Tried to Scan after Itto and Gorou's banner and it's happening the same to me. It runs until the end, but the json file isn't created.


WIP: Adds data aggregation parameterized routes

What
Added handling for parameterized routes allowing using topic  and subtopic titles. This will allow accessing pages such as:
https://www.ons.gov.uk/economy/publications
https://www.ons.gov.uk/economy/grossdomesticproductgdp/publications
https://www.ons.gov.uk/businessindustryandtrade/publications
https://www.ons.gov.uk/businessindustryandtrade/business/publications
etc.
This is a WIP. The feature itself works but it uses the Topic API for each check. A general topic cache must be created that gets used for these checks. I have added the first steps of creating it in the last commit.
How to review
Requires: https://github.com/ONSdigital/dp-frontend-router/pull/448
Who can review
Anyone

Closed due to stemming from the wrong branch. Fixed in #284


Run tests against python 3.6 and switch some places where a default p…

…ython is used from py27 to py36

Coverage remained the same at 100.0% when pulling b0eea2134a45044d0bf9d213e353a6f040a46cae on run_tests_with_py36 into 2c88188abb51c345b23f5ea50d7f2de0b6b50cf0 on master.


Issue#68 Removed System.exit() calls

Closes #68
Removed and/or replaced the System.exit() calls from the code.

Hi, thank you for your pull request.
I think you misunderstood the linked issue. The linked issue suggests adding a new class that finds all invocations of System.exit and then emits a message that discourages the use of this. Not removing the System.exit calls in this project (which are required, because the exit code of a program other than 0, can only be set through it).
I assume that because you linked a good first issue that you want to contribute to this project, which is why I will explain in the following a bit more about how you can implement this lint.
This tool is supposed to annotate certain mistakes in code for grading.
You implement those a separate classes that search in the code for that mistake. For example CompareObjectsNotStringsCheck searches for all instances in the code, where a method is called. It then checks if one uses equals via toString instead of implementing it correctly:
if (a.toString().equals(b.toString()) {
}
// instead of
if (a.equals(b) {
}

For implementing the linked issue, you can start by copying the CompareObjectsNotStringsCheck of this project. To find all System.exit calls in the code, you declare that you want to process CtInvocations:
staticAnalysis.processWith(new AbstractProcessor<CtInvocation<?>>() {
    @Override
    public void process(CtInvocation<?> invocation) {
         // here you need to figure out if that invocation variable is a call to `System.exit`
    }
});

You would also have to add a new variant to the ProblemType enum for your new class and add a helpful message in strings.de.ftl and strings.en.ftl.
For tests you can look at https://github.com/Feuermagier/autograder/tree/main/autograder-core/src/test/resources/de/firemage/autograder/core/check_tests. In the config.txt you declare on which line a message should happen (so in every line where System.exit is called).
If you have any further questions, feel free to ask.


Adding "auth" and "auth/gcp" packages

As devs move away from marvin and App Engine 1st generation and back to the land of gizmo with the 2nd generation, we need some tools to replace GAE's auth mechanisms for internal and external traffic. This is the first iteration of tools to fill this need.
The auth package provides a new interface for servers who will be verifying inbound signed JWTs. This is meant to be the server side version of oauth2.TokenSource
type PublicKeySource interface {
    Get(context.Context) (PublicKeySet, error)
}

Two major implementations are being added, one for IAM and one for Google identity tokens. These are supported by helper functions for parsing JWKS responses from URL or directly from JSON.
To make things easier on the users, the auth package is also introducing a new Verifier type, that will be composed of a PublicKeySource and helper functions for decoding and verifying inbound tokens.
type Verifier struct {
    ks PublicKeySource
    df ClaimsDecoderFunc
    vf VerifyFunc
}

For the client side of things, the gcp package also introduces an oauth2.TokenSource for IAM and the Identity token.
There are a handful of other handy functions and tools to help tie everything together. I hope to add more soon that will reduce the amount of (already minimal) configuration required along with tools to make local and test environments easer to work with.
Both IAM and Identity are being provided because we're prefer to use Identity tokens in GCP, but they aren't available in the 1st generation runtime. The IAM tools provided are meant to be a bridge for users migrating large projects over to the 2nd generation runtime, one service at a time.
Side note: I decided to go against using other JWT libraries as they have been taking a more generic approach and allow for multiple forms of crypto. We're just using the rsa.PublicKey in both of our use cases so I took a more opinionated approach.

Coverage remained the same at 44.895% when pulling c0c17b42c3cc4a6c53d9fdb9129d13ea56f201b0 on auth into bb17238799b62b40a3bcb83a072d980562071436 on master.


About the data of endipointerror avg

Thank you for your last reply. I successfully reproduced the endpointerror values of Control and CSRT. The trend of the curve is similar to the figure in the SIRT paper, but the value is 2.2-2.3 times higher than that in the paper. I think Do you know if other processing has been done? Below is my code for numerical processing.
data_csrt = {}
data_mft ={}
for filename in os.listdir(folder_path_csdt):
    if filename.endswith('.json'):
        file_path = os.path.join(folder_path_csdt, filename)
        with open(file_path, 'r') as file:
            json_data = json.load(file)
            data_csrt.update(json_data)

for filename in os.listdir(floder_path_mft):
    if filename.endswith('.json'):
        file_path = os.path.join(floder_path_mft, filename)
        with open(file_path, 'r') as file:
            json_data = json.load(file)
            data_mft.update(json_data)

x = []
y = []
x1 = []
y1 = []
y_csrt = []
y_mft = []
for key, value in data_csrt.items():
    if key != "total":
        x.append(value["endpointerror_videotime"])
        y.append(value["endpointerror_control"])
        y_csrt.append(value["endpointerror_CSRT"])

for key, value in data_mft.items():
    if key != "total":
        x1.append(value["endpointerror_videotime"])
        y1.append(value["endpointerror_control"])
        y_mft.append(value["endpointerror_MFT"])

sorted_data = sorted(zip(x, y, y_csrt), key=lambda d: d[0])
sort_mft_data = sorted(zip(x1, y1, y_mft), key=lambda d: d[0])
x_sorted, y_sorted, y_csrt_sorted = zip(*sorted_data)
x_1, y_1, y_mft_sorted = zip(*sort_mft_data)

data_avg_con = []
data_avg_csrt = []
data_avg_mft = []

sum_control = 0
sum_CSRT = 0
sum_MFT =0

for i in range(len(x_sorted)):
    if x_sorted[i] >=10:
        break
    sum_control = sum_control + (y_sorted[i])
    avg_con = sum_control / ((i+1))

    sum_CSRT = sum_CSRT + (y_csrt_sorted[i])
    avg_csrt = sum_CSRT / ((i+1))

    sum_MFT = sum_MFT + (y_mft_sorted[i])
    avg_mft = sum_MFT / ((i+1))

    data_avg_con.append(avg_con)
    data_avg_csrt.append(avg_csrt)
    data_avg_mft.append(avg_mft)

x_sorted = x_sorted[0:i]

plt.figure(figsize=(18, 6))
# plt.scatter(x_sorted, data_avg, color='red', label='endpointerror_control', s=5)
# plt.scatter(x_sorted, y_csrt_sorted, color='red', label='endpointerror_MFT', s = 5)
plt.plot(x_sorted, data_avg_con, color='red', label='endpointerror_Control')
plt.plot(x_sorted, data_avg_csrt, color='green', label='endpointerror_CSRT')
plt.plot(x_sorted, data_avg_mft, color='blue', label='endpointerror_MFT')‘

This is the experimental result I ran. I also want to ask if you will disclose the code of the SENDD model. If so, it will be of great significance for us to participate in the challenge.

At a glance, the code looks good to me. The codebase here is a re-write of that in the paper, so there are bound to be some differences (along with a less filtered dataset). We also average over number of tracks, not videos in the paper from what I remember. Both of these together could add to a factor of difference. That all said, a factor of ~2 is large, so I'll have to dig up the old code and look through it (might be something like a scale change from initial data to that which we released).
For now, the relative ordering from your figure looks good (as in MFT<CSRT<Control), so evaluating comparative performance should be fine.
Regarding SENDD, we will be unable to release the code for this framework, but we note that since SENDD is a framewise tracker (scene flow) without drift correction, we expect methods that re-localize on top of framewise trackers, such as MFTs, to outperform on this challenge.

I'm going to close this for now since there are other tasks that are taking up my time for the STIR challenge preparation in the meantime.


Use OS port selection for the local OAuth server

Is your feature request related to a problem? Please describe.
In the OAuthClient, we start a local HTTP server by trying out a series of ports. However the port selection could be made by the operating system by passing 0 as port number.
Describe the solution you'd like
from http.server import HTTPServer, SimpleHTTPRequestHandler
server = HTTPServer(("<IP_ADDRESS>", 0), SimpleHTTPRequestHandler)
print(server.server_address, server.server_name, server.server_port)

Sounds like a good idea. I am going to mark this one as a good first issue.

from http.server import HTTPServer, SimpleHTTPRequestHandler
server = HTTPServer(("<IP_ADDRESS>", 0), SimpleHTTPRequestHandler)
print(server.server_address, server.server_name, server.server_port)

from http.server import HTTPServer, SimpleHTTPRequestHandler
server = HTTPServer(("<IP_ADDRESS>", 0), SimpleHTTPRequestHandler)
print(server.server_address, server.server_name, server.server_port)


Get comments from doc-xml for constructor parameters.

Right now, comments only get comments from simple comments right before arguments.
Related issue: #2

Fixed in 1.0.16

Thank you, highly appreciated!


Highlight the line number where the cursor locates

VSCode Version: 1.5.2
OS Version: Windows 10 Anniversary

Steps to Reproduce:

Can we have a feature that highlights the line number when the cursor locales in some line which helps to find where my cursor is?

@alexandrudima this request seems fixed in version 1.8


Modal form submit

we want to keep the browser behaviour, so when user submits the form inside of an iframe by any means it triggers the click on the button, and if the button is the default one we trigger the default behaviour with the hiding frame, reloading, checking that.saved etc.

LGTM

Coverage remained the same at 89.333% when pulling 84d2864f43f631a84e691562e169426d4ef70114 on FinalAngel:feature/modal-form-submit into 987a4368c41f09506a74773fe44e1a89d69d148b on divio:develop.


Update README.md

A data science/machine learning YouTube channel was added

A data science/machine learning YouTube channel was added

مرسی بابت معرفی 🙏

مرسی از شما بابت ساخت این ریپو عالی.
On Tue, 17 Oct 2023 at 11:52, Reza Shakeri @.***> wrote:

A data science/machine learning YouTube channel was added
مرسی بابت معرفی 🙏
—
Reply to this email directly, view it on GitHub
https://github.com/barnamenevisi/free-resources/pull/8#issuecomment-1765912932,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ATOAFKUI536KWNKEHSZAX2DX7Y53DAVCNFSM6AAAAAA6DMZZOKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRVHEYTEOJTGI
.
You are receiving this because you authored the thread.Message ID:
@.***>


original pytorch or onnx model

Could you please provide pretrained pytorch or onnx model weights also? I noticed you only shared converted ncnn models, but I would like to see the speed of inference on gpu/npu accelerated systems

Actually the download link is in readme. Here is the link https://drive.google.com/file/d/1EhMqGozKfqEfw8y9ftbi1jhYu86XoW62/view?usp=sharing

That is the torch jit model though, demo.py will not accept that. or am I doing something wrong?
I believe you need to upload the checkpoint weights. Not torch jit weights.

This .pth file is saved with torch1.7. Pytorch have changed .pth format after 1.6. You can update your pytorch version. I may upload an old format later.

Oh thats why then. I was trying it with 1.3 as requested in your requirements.txt

This .pth file is saved with torch1.7. Pytorch have changed .pth format after 1.6. You can update your pytorch version. I may upload an old format later.

Would it be possible to share the simplified onnx model perhaps then? Due to the restrictions of the SOC I am working at, I cannot use torch models above 1.2.0

This .pth file is saved with torch1.7. Pytorch have changed .pth format after 1.6. You can update your pytorch version. I may upload an old format later.

Would it be possible to share the simplified onnx model perhaps then? Due to the restrictions of the SOC I am working at, I cannot use torch models above 1.2.0

Try this .pth file. It's for torch<1.6.  https://drive.google.com/file/d/10h-0qLMCgYvWQvKULqbkLvmirFR-w9NN/view?usp=sharing


Set k8s built-in labels through the CCM

https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/
We should be setting some of those built-in labels like node.kubernetes.io/instance-type and topology.kubernetes.io/zone so that they match with packet naming.

Turns out we already did. Here is output from one node:
metadata:
  annotations:
    kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
    node.alpha.kubernetes.io/ttl: "0"
    packet.com/node.asn: "65000"
    packet.com/peer.asns: "65530"
    packet.com/peer.ips: <IP_ADDRESS>
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
  creationTimestamp: "2020-09-03T14:04:10Z"
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/instance-type: baremetal_0
    beta.kubernetes.io/os: linux
    failure-domain.beta.kubernetes.io/region: ewr1
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: master-01
    kubernetes.io/os: linux
    node-role.kubernetes.io/master: ""
    node.kubernetes.io/instance-type: baremetal_0
    topology.kubernetes.io/region: ewr1

The last two lines are the ones we need. The region is correct, but the instance-type is wrong. PR coming shortly.


fixed bug in documentation

fl_sector and fl_data_off do not exist. Probably should be fe_sector and fe_data_off.

OK, done.


commit deeplabv3+_CamVid_data_parallel

An example for training deeplabv3+ on CamVid data set

Hi @JaclynZhong1127 , thanks for your pull request! We will review it and let you know soon.

Hi, can you remove the images and tensorboard logs from your commit? These will make the git file bulky. Meanwhile, we have shifted the examples in the new repository. Would you mind put up a PR there? I will close this PR for now. Feel free to raise any issue in the new examples repository if you encounter any problem.


Cuda matrix conversion csr sellp

Reference and cuda conversion routines between csr and sellp formats

Same here, after merging #225 you have a conflict so you need to update.
The most probable cause is that you add kernels from all of these PRs into CSR, which mean conflicting changes in terms of where the kernels are declared.

@fritzgoebel if you update your PRs to the most recent develop, you will benefit from lighter archives in your pipeline which (should hopefully) make the CI pass.


Support bulk publish in Azure Service Bus

Describe the feature
Support Bulk Publish API in Azure Service Bus.
Release Note

RELEASE NOTE: ADD Bulk publish support in Azure Service Bus

Closed by #2106


Fixed AlgorithmIdenifier.parameters of digest algo, extended ECIES, added XIES

This is an amendment to the pull #905.

Sorry to do this with late notice, but would you split this one up? If I could get one PR for XDHBasicAgreenent, one for XIES, it would be a lot easier to deal with this. I can sort out the AlgorithmIdentifier changes from here. Thanks.

Just created two pull requests #961 and #962. Because the XIES depends on the patch of ECIES (962), I will upload the PR for XDH after the merge of ECIES.

Closing as AlgorithmIdentifier changes should now be done.


Api routes can't be accessed even they are listed in api:routes

| Q                 | A
| Api routes can't be accessed even they are listed in api:routes
| Bug?              | not sure
| New Feature?      | no
| Framework         | Laravel
| Framework version | 5.4
| Package version   | 1.0.*@dev
| PHP version       | 5.6
Actual Behaviour
when i use api:routes to list all the api routes in console, the routes are all there, the question is
that some routes can be access and return the expected data while others can't be accessed and return something like
{"message":"404 Not Found","status_code":404,"debug":{"line":179,"file":"/Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Routing/RouteCollection.php","class":"Symfony\Component\HttpKernel\Exception\NotFoundHttpException","trace":["#0 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Routing/Router.php(546): Illuminate\Routing\RouteCollection->match(Object(Dingo\Api\Http\Request))","#1 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Routing/Router.php(525): Illuminate\Routing\Router->findRoute(Object(Dingo\Api\Http\Request))","#2 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Routing/Router.php(511): Illuminate\Routing\Router->dispatchToRoute(Object(Dingo\Api\Http\Request))","#3 /Users/liaolliso/work/laravel54/vendor/dingo/api/src/Routing/Adapter/Laravel.php(81): Illuminate\Routing\Router->dispatch(Object(Dingo\Api\Http\Request))","#4 /Users/liaolliso/work/laravel54/vendor/dingo/api/src/Routing/Router.php(513): Dingo\Api\Routing\Adapter\Laravel->dispatch(Object(Dingo\Api\Http\Request), 'v1')","#5 /Users/liaolliso/work/laravel54/vendor/dingo/api/src/Http/Middleware/Request.php(126): Dingo\Api\Routing\Router->dispatch(Object(Dingo\Api\Http\Request))","#6 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(114): Dingo\Api\Http\Middleware\Request->Dingo\Api\Http\Middleware\{closure}(Object(Dingo\Api\Http\Request))","#7 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(30): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Dingo\Api\Http\Request))","#8 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Foundation\Http\Middleware\TransformsRequest->handle(Object(Dingo\Api\Http\Request), Object(Closure))","#9 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(30): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Dingo\Api\Http\Request))","#10 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Foundation\Http\Middleware\TransformsRequest->handle(Object(Dingo\Api\Http\Request), Object(Closure))","#11 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Dingo\Api\Http\Request))","#12 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Foundation\Http\Middleware\ValidatePostSize->handle(Object(Dingo\Api\Http\Request), Object(Closure))","#13 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/CheckForMaintenanceMode.php(46): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Dingo\Api\Http\Request))","#14 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Foundation\Http\Middleware\CheckForMaintenanceMode->handle(Object(Dingo\Api\Http\Request), Object(Closure))","#15 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(102): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Dingo\Api\Http\Request))","#16 /Users/liaolliso/work/laravel54/vendor/dingo/api/src/Http/Middleware/Request.php(127): Illuminate\Pipeline\Pipeline->then(Object(Closure))","#17 /Users/liaolliso/work/laravel54/vendor/dingo/api/src/Http/Middleware/Request.php(103): Dingo\Api\Http\Middleware\Request->sendRequestThroughRouter(Object(Dingo\Api\Http\Request))","#18 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Dingo\Api\Http\Middleware\Request->handle(Object(Illuminate\Http\Request), Object(Closure))","#19 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))","#20 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(102): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))","#21 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(151): Illuminate\Pipeline\Pipeline->then(Object(Closure))","#22 /Users/liaolliso/work/laravel54/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(116): Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter(Object(Illuminate\Http\Request))","#23 /Users/liaolliso/work/laravel54/public/index.php(54): Illuminate\Foundation\Http\Kernel->handle(Object(Illuminate\Http\Request))","#24 {main}"]}}
The difference is that:
$api->get('keye_route/{id}', ['as' => 'keye_route.get', 'uses' => 'KeyeRouteController@get'])->where('id', '[0-9]+'); // can be accessed, example: /api/keye_route/1232131231
$api->get('keye_routes/get_by_tag', ['as' => 'keye_route.get_by_tag', 'uses' => 'KeyeRouteController@getTagRoutes']);  // can't be accessed, example: /api/keye_route/get_by_tag?tag=aaa&page=1
In controller i use $id for the first route to get params while i use Request $request to get the params in second route.
More: i can access them some days ago while i can't now even without any touch to the code.  I wonder if it's because of composer update which i did recently, im not sure.
Hope someone could help. Thanks
Expected Behaviour
api routes could be access like other routes
Steps to Reproduce
Possible Solutions
If you have any ideas on how to solve the issue, add them here, otherwise you can omit this part.

idiot i am.... lose a s in the route


Does not build on 32bit OS

Just tried compiling FastRTPS (HEAD at fa77f1319e7b37869b66a29e47ea1cf4916260fe) from source on Raspberry Pi 4 (Raspbian Release 10, Buster) and it fails on
[  1%] Building CXX object src/cpp/CMakeFiles/fastrtps.dir/rtps/persistence/SQLite3PersistenceService.cpp.o
/home/pi/projects/third_party/Fast-RTPS/src/cpp/rtps/persistence/SQLite3PersistenceService.cpp: In member function ‘virtual bool eprosima::fastrtps::rtps::SQLite3PersistenceService::add_writer_change_to_storage(const string&, const eprosima::fastrtps::rtps::CacheChange_t&)’:
/home/pi/projects/third_party/Fast-RTPS/src/cpp/rtps/persistence/SQLite3PersistenceService.cpp:182:78: error: ‘const struct eprosima::fastrtps::rtps::SequenceNumber_t’ has no member named ‘to64long’
  182 |         sqlite3_bind_int64(add_writer_change_stmt_, 2, change.sequenceNumber.to64long());
      |                                                                              ^~~~~~~~
/home/pi/projects/third_party/Fast-RTPS/src/cpp/rtps/persistence/SQLite3PersistenceService.cpp: In member function ‘virtual bool eprosima::fastrtps::rtps::SQLite3PersistenceService::remove_writer_change_from_storage(const string&, const eprosima::fastrtps::rtps::CacheChange_t&)’:
/home/pi/projects/third_party/Fast-RTPS/src/cpp/rtps/persistence/SQLite3PersistenceService.cpp:213:81: error: ‘const struct eprosima::fastrtps::rtps::SequenceNumber_t’ has no member named ‘to64long’
  213 |         sqlite3_bind_int64(remove_writer_change_stmt_, 2, change.sequenceNumber.to64long());
      |                                                                                ^~~~~~~~
/home/pi/projects/third_party/Fast-RTPS/src/cpp/rtps/persistence/SQLite3PersistenceService.cpp: In member function ‘virtual bool eprosima::fastrtps::rtps::SQLite3PersistenceService::update_writer_seq_on_storage(const string&, const eprosima::fastrtps::rtps::GUID_t&, const eprosima::fastrtps::rtps::SequenceNumber_t&)’:
/home/pi/projects/third_party/Fast-RTPS/src/cpp/rtps/persistence/SQLite3PersistenceService.cpp:270:63: error: ‘const struct eprosima::fastrtps::rtps::SequenceNumber_t’ has no member named ‘to64long’
  270 |         sqlite3_bind_int64(update_reader_stmt_, 4, seq_number.to64long());
      |                                                               ^~~~~~~~
make[2]: *** [src/cpp/CMakeFiles/fastrtps.dir/build.make:1740: src/cpp/CMakeFiles/fastrtps.dir/rtps/persistence/SQLite3PersistenceService.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:987: src/cpp/CMakeFiles/fastrtps.dir/all] Error 2
make: *** [Makefile:163: all] Error 2

Hi @cvilas,
Could you please test if removing the guard lines SequenceNumber.h:67 and SequenceNumber.h:75?
We think that checking support is not needed anymore and should solve the issue directly.

@LuisGP I can confirm that compilation succeeds after removing the guards.
lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 10 (buster)
Release:        10
Codename:       buster

gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/8/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Raspbian 8.3.0-6+rpi1' --with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=arm-linux-gnueabihf- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --disable-libquadmath-support --enable-plugin --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --disable-werror --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1)

Thank you very much.
We'll create a hotfix with the changes too.


fix(ci): properly use docker save to store images.

What type of PR is this?
/kind bug
Any specific area of the project related to this PR?
/area CI
What this PR does / why we need it:
Normal (non buildx) docker does not support direct storing to tar.
Which issue(s) this PR fixes:
CI.
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

NONE

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cc @LucaGuerra

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, LucaGuerra
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, jasondellaluce, LucaGuerra
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,jasondellaluce]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Minor improvements to the README

Two minor additions to the README:

Add conference talk from John and Natalia at eBPF day 2022
Add a small pointer to the development setup in the FAQ

The CI failure is unrelated, merging!


migrate to uproxy-lib v27, enable obfuscation

This moves to uproxy-lib v27's BridgingPeerConnection which provides support for negotiating obfuscated peerconnections. This is backwards compatible except for one-time connections: If we know a client is running with MESSAGE_VERSION 2, initiate an obfuscated connection; for older clients, same as today.
There's one outstanding thing missing from churn: support for generating a random key for each connection. I will work on that, it shouldn't change this pull request.
Tested on Chrome with the Google+ social provider and one-time connections.
Question: I had to re-order the STUN servers in order to make churn work from my internal network...should we bump the STORAGE_VERSION too?

@bemasc, FYI.

Random key generation for churn is under review:
https://github.com/uProxy/uproxy-lib/pull/183

Review status: 0 of 18 files reviewed, 3 unresolved discussions, some commit checks failed.

src/generic_core/remote-connection.ts, line 77 [r2] (raw file):
Can we change the remoteVersion arg to something more meaningful like useBridge :boolean

src/generic_core/remote-connection.ts, line 154 [r2] (raw file):
Can we change remoteVersion to just a boolean here like useObfuscation :boolean"?  Then I think it would be simpler to change this if we decide to make obfuscation a setting

src/generic_core/remote-instance.ts, line 194 [r2] (raw file):
I think changing this logic is risky and I'd rather keep the old logic if its not hard to do.  In the old logic we would always reset the peer connection each time we got an offer, which means that we won't get into a broken state if a connection is attempted but fails (failing in a way that rtcToNet_.onceStopped isn't called, which I think is the case if the connection never succeeded in the first place)

Comments from the review on Reviewable.io

OK, I tested this on Firefox. There are still some issues with churn.
As discussed, let's just enable the bridge for now -- and hold off on obfuscation.
I've changed MESSAGE_VERSION = 2 to mean: use bridge, without obfuscation. Tested on Firefox and this seems to work just fine.

Review status: 0 of 18 files reviewed, 3 unresolved discussions, some commit checks pending.

src/generic_core/remote-connection.ts, line 77 [r2] (raw file):
That's an option, and I can do that but since the number of options should grow over time I think you really want a function somewhere named something like remoteVersionToBridge which knows what sort of bridge to create, given a version number.
With this pull request I was kinda just trying to make things work, though.

src/generic_core/remote-connection.ts, line 154 [r2] (raw file):
See previous comment.

src/generic_core/remote-instance.ts, line 194 [r2] (raw file):
OK. I'm still not sure this isn't something the UI should set in social messages but it's straightforward to add in the backend, and might be useful there for copypaste, too. See this pull request:
https://github.com/uProxy/uproxy-lib/pull/186
This should no longer be risky.

Comments from the review on Reviewable.io

Depends on this pull request:
https://github.com/uProxy/uproxy-lib/pull/186

Review status: 0 of 18 files reviewed, 3 unresolved discussions, some commit checks pending.

Comments from the review on Reviewable.io

:+1:

Review status: 0 of 18 files reviewed, all discussions resolved, some commit checks failed.

Comments from the review on Reviewable.io


Failed ChromeOS jobs on April 8th

One example:
https://github.com/ClangBuiltLinux/continuous-integration2/actions/runs/4648117706
Seems like another caching run is needed?
cc @vishalbhoj

Hi Nathan,
Yes this is again a caching issue. The chromiumos kernel tree is the largest and with git operations the disk usage grows very quickly. I have cleaned up the cache again to make it functional. I'll work on handling it better and post back on this issue.

I have not seen this occur recently, so I am closing this up for now.


Provide prebuilt artifacts for additional architecture (aarch64)

What is the problem you're trying to solve
https://download.docker.com is pretty slow to release new versions and doesn't provide all versions
I would like to make kubespray use the prebuilt versions from https://github.com/containerd/containerd/releases instead of docker repo, but sadly linux aarch64 build are missing
Describe the solution you'd like
Have linux aarch64 build for all new versions
Additional context

We initially added release binaries for arm64 in #5329 but after it blocked our ability to have successful releases due to some complexities in the release build process, we had to revert for now in #5437 while we got the 1.5.x release successfully out the door.  At this point it would be good to go back and figure this out now that the pressure is off for getting a release out. We are running a full CI test suite on arm64 now (#5406) so it's just a matter of fixing the release issue(s).

We have been providing nightly builds on arm64 for awhile if that's of any value in the short term before the next release: https://github.com/containerd/containerd/actions/runs/862194458

Thanks a lot for the detailed answer, I'm more looking for the long term to have a known fully supported build of containerd for kubespray (https://github.com/kubernetes-sigs/kubespray) (and so is @floryut)

kind-ci  already provides arm binaries (not RPM/dpkg, though): https://github.com/kind-ci/containerd-nightlies/releases/tag/containerd-1.5.2

@estesp @AkihiroSuda any chance we can get some traction on this? I'd be happy to try and lend a hand.
In kubespray we are considering to make containerd the default container manager and are in the process of moving from docker or distro provided binaries to using your build artifacts. While we have developers in favor of this change, it would remove our support for AArch64 in the default configuration.

kind-ci already provides arm binaries

@AkihiroSuda Those are only containerd binaries though. The releases in containerd/containerd include the full package that's referred in the installation guides, i.e. cri-containerd-cni-1.5.8-linux-amd64.tar.gz.
@estesp Any way I can help with re-integrating the release process? Would adding it to matrix here and use it in the rest of the steps help? Maybe even just backport that file from main branch where it seems to cover all platforms?

FWIW, I was able to get a release built in my fork with the related changes in main: https://github.com/muvaf/containerd/releases/tag/v1.5.8

1.6 releases (in beta now) already have added ARM64 via another path--moving all release building inside a Dockerfile via #6196 ; so we are all set with the aarch64 support in 1.6.0 and above.
I'm not sure how tricky it might be (or whether we want to) add that to the 1.5.x release scripting. We now run full unit and integration testing on ARM64/aarch64 on each PR, so that gave us the confidence to add the ARM64/aarch64 binaries to our official releases page.
I see from a recent release/1.5 PR I see we are running that same level of testing on 1.5.x, so maybe we can consider that same update to the release scripting but that would need to be validated and agreed to among the maintainers.

@champtar Could this be closed by now?


Documentation Assistance [Documentation]

What problem does this feature solve?
I am trying to exctract information from the documentation that would look something like this:
{
"type": {
"Format": "LabelList",
"Description": "Type of chart.",
"Example": "bar",
"Labels": ["line", "bar", "pie", "scatter", "effectScatter", "radar", "tree", "treemap", "sunburst", "boxplot", "candlestick", "heatmap", "map", "parallel", "lines", "graph", "sankey", "funnel", "gauge", "pictorialBar", "themeRiver", "custom"],
"Link": "https://echarts.apache.org/en/option.html#series-bar.type",
ChartType: [line, scatter,...]
},
or
"label.position": {
"Format": "LabelList",
"Description": "Position of the label.",
"Example": "inside",
"Labels": ["top", "left", "right", "bottom", "inside", "insideLeft", "insideRight", "insideTop", "insideBottom", "insideTopLeft", "insideTopRight", "insideBottomLeft", "insideBottomRight"],
"Link": "https://echarts.apache.org/en/option.html#series-bar.label.position"
 ChartType: [line, scatter,...]

},
An extraction of this information would be invaluable to me if someone has already done something like this or if the developers might be able to assist.
What does the proposed API look like?
This information would allow me to develop a subset of options in a developers user interface based on what they are working on and limit the possibility or errors and looking through the website to find the address of the property they are trying to change.
I have tried ChatGPT, Claude, WebCrawlers and more to try and extract the data, but it is not accurate enough for me.
Please help :-)

Note that the Formats that I am personally after are
Value
Percent
Boolean
Color
String
LabelList (a string with a predefined set of applicable strings)
Anything like this would be very helpful.


Expose 'download_options' parameter on Archive resource when downloading installer to support custom curl download options

Going from v1.4.0 to v.1.6.0 was a breaking change for us because the default for allow_insecure was changed from true to false. While the parameter is provided for switching allow_insecure to false, this has poor security implications and has been a known issue for us. We have this issue for all our Puppet that use the Archive Puppet Forge module.
The Archive resource uses its own curl distribution rather than the OS distribution (Curl is default provider on Linux and therefore RedHat). The Puppet curl does not use the system-wide CA certificates and will fail if the user deploys their own internal PKI management.
Error: Execution of '/opt/puppetlabs/puppet/bin/curl https://*/e/*/api/v1/deployment/installer/agent/unix/default/latest/?Api-Token=*&arch=all -o /tmp/Dynatrace-OneAgent-Linux-latest.sh_20210218-15780-7h9ij3 -fsSL --max-redirs 5' returned 60: curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Error: /Stage[main]/Dynatraceoneagent::Download/Archive[Dynatrace-OneAgent-Linux-latest.sh]/ensure: change from 'absent' to 'present' failed: Execution of '/opt/puppetlabs/puppet/bin/curl https://dynatrace.np.costco.com/e/5c210202-713c-45a3-9d09-e20ca3e19b17/api/v1/deployment/installer/agent/unix/default/latest/?Api-Token=hdUZy8BiQ0Sm1N8e8id-A&arch=all -o /tmp/Dynatrace-OneAgent-Linux-latest.sh_20210218-15780-7h9ij3 -fsSL --max-redirs 5' returned 60: curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above. (corrective)
Info: Class[Dynatraceoneagent::Install]: Unscheduling all events on Class[Dynatraceoneagent::Install]
Notice: /Stage[main]/Dynatraceoneagent::Install/Exec[install_oneagent]: Dependency Archive[Dynatrace-OneAgent-Linux-latest.sh] has failures: true

Resolution:
Starting with v2.0.0, the Archive resource provides a 'download_options' parameter that can be used to specify the CA certificate bundle to use. We resolved this issue by pathing the module to specify the root CA chain in the Archive resource for downloading the agent installer. The parameter takes an array of strings.
Archive {
    download_options => ['--cacert', '/etc/pki/tls/certs/ca-bundle.crt'],
}

download_options added on release 1.7.0


sprint_statistics() method gives KeyError: 'test_loss' in 0.2 version

I got an error while calling sprint_statistics() in autopytorch version 0.2
model.sprint_statistics()

error:
...
cost_info = run_value.additional_info[f'{inference_name}_loss']
KeyError: 'test_loss'

Could you provide the code to reproduce this error?

@ravinkohli  Thank you for your quick response, the below are the code snipt, the same code i was using in the previous version (0.1.1)
model = TabularClassificationTask(
                    ensemble_size=5,
                    ensemble_nbest=5,
                    max_models_on_disc=5
                )
model.search(
                X_train=X_train,
                y_train=y_train,
                total_walltime_limit=100,
                memory_limit=4072,
                enable_traditional_pipeline=False
            )
ss = model.sprint_statistics()  # Error is on this statement, In 0.1.1 version it  is working but 0.2 version it throws error.
print(ss)

Okay, I see. Thanks for pointing out this issue. We'll release a version with a bug fix for this ASAP. If you can, you can use version 0.1.1 till then. Or,  you can simply pass X_train and y_train to the test data as well, because I think the issue is that model.sprint_statistics() now assumes that there will always be test data.

Okay, I see. Thanks for pointing out this issue. We'll release a version with a bug fix for this ASAP. If you can, you can use version 0.1.1 till then. Or, you can simply pass X_train and y_train to the test data as well, because I think the issue is that model.sprint_statistics() now assumes that there will always be test data.

Okay, I see. Thanks for pointing out this issue. We'll release a version with a bug fix for this ASAP. If you can, you can use version 0.1.1 till then. Or, you can simply pass X_train and y_train or some of it to the test data as well, because I think the issue is that model.sprint_statistics() now assumes that there will always be test data.

Test data will not interfere with the optimisation process, it will only add to the time it takes for a configuration as we will also predict using the test data. Sorry for the inconvenience.

Hi, we have fixed this issue in the latest release.  You can install it using pip install autoPyTorch==0.2.1. I am closing this issue for now.


HBASE-28527: Adjust BlockCacheKey to use the file path instead of file name.

The time-based priority eviction policy relies on the presence of path in
the BlockCacheKey to fetch the required metadata to check data hotness and
decide whether or not to retain the block in the bucket cache.
Hence, the constructor of BlockCacheKey is adjusted to take the file path
as the input parameter. The code paths that create the blockCacheKey and
also the unit tests need to be adjusted to pass the path instead of file name.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 41s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ HBASE-28463 Compile Tests _

+1 :green_heart:
mvninstall
4m 55s
HBASE-28463 passed

+1 :green_heart:
compile
3m 20s
HBASE-28463 passed

+1 :green_heart:
checkstyle
0m 41s
HBASE-28463 passed

+1 :green_heart:
spotless
0m 52s
branch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
1m 53s
HBASE-28463 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 58s
the patch passed

+1 :green_heart:
compile
3m 15s
the patch passed

+1 :green_heart:
javac
3m 15s
the patch passed

-0 :warning:
checkstyle
0m 46s
hbase-server: The patch generated 10 new + 85 unchanged - 0 fixed = 95 total (was 85)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
6m 14s
Patch does not cause any errors with Hadoop 3.3.6.

-1 :x:
spotless
1m  8s
patch has 69 errors when running spotless:check, run spotless:apply to fix.

+1 :green_heart:
spotbugs
2m 45s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 16s
The patch does not generate ASF License warnings.

40m 38s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5832/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5832

JIRA Issue
HBASE-28527

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile

uname
Linux 14b3ea56ed94 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
HBASE-28463 / b7bb8b9d27

Default Java
Eclipse Adoptium-11.0.17+8

checkstyle
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5832/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt

spotless
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5832/1/artifact/yetus-general-check/output/patch-spotless.txt

Max. process+thread count
77 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5832/1/console

versions
git=2.34.1 maven=3.8.6 spotbugs=4.7.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 43s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ HBASE-28463 Compile Tests _

+1 :green_heart:
mvninstall
4m 54s
HBASE-28463 passed

+1 :green_heart:
compile
1m  9s
HBASE-28463 passed

+1 :green_heart:
shadedjars
6m 55s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 44s
HBASE-28463 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 54s
the patch passed

+1 :green_heart:
compile
0m 57s
the patch passed

+1 :green_heart:
javac
0m 57s
the patch passed

+1 :green_heart:
shadedjars
6m 33s
patch has no errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
0m 27s
hbase-server generated 1 new + 96 unchanged - 0 fixed = 97 total (was 96)

_ Other Tests _

+1 :green_heart:
unit
234m 25s
hbase-server in the patch passed.

265m  3s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5832/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5832

JIRA Issue
HBASE-28527

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 33900618b5c9 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
HBASE-28463 / b7bb8b9d27

Default Java
Eclipse Adoptium-11.0.17+8

javadoc
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5832/1/artifact/yetus-jdk11-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5832/1/testReport/

Max. process+thread count
5198 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5832/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 57s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ HBASE-28463 Compile Tests _

+1 :green_heart:
mvninstall
2m 38s
HBASE-28463 passed

+1 :green_heart:
compile
0m 42s
HBASE-28463 passed

+1 :green_heart:
shadedjars
5m 10s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 26s
HBASE-28463 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 25s
the patch passed

+1 :green_heart:
compile
0m 42s
the patch passed

+1 :green_heart:
javac
0m 42s
the patch passed

+1 :green_heart:
shadedjars
5m 10s
patch has no errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
0m 25s
hbase-server generated 1 new + 22 unchanged - 0 fixed = 23 total (was 22)

_ Other Tests _

+1 :green_heart:
unit
245m 20s
hbase-server in the patch passed.

268m 44s

Subsystem
Report/Notes

Docker
ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5832/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5832

JIRA Issue
HBASE-28527

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 8b82e32a7389 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
HBASE-28463 / b7bb8b9d27

Default Java
Temurin-1.8.0_352-b08

javadoc
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5832/1/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5832/1/testReport/

Max. process+thread count
5281 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5832/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

The time-based priority eviction policy relies on the presence of path in
the BlockCacheKey to fetch the required metadata to check data hotness and
decide whether or not to retain the block in the bucket cache.

Can you explain why do you need that? This is not a simple/small change.

The time-based priority eviction policy relies on the presence of path in
the BlockCacheKey to fetch the required metadata to check data hotness and
decide whether or not to retain the block in the bucket cache.

Can you explain why do you need that? This is not a simple/small change.

Our design originally relied on the presence of path in the BlockCacheKey so that we parse the path to get to the regionID and column family to reach to the file and access its metadata. (The framework that @vinayakphegde  has implemented).
The purpose of using the path is slightly avoid the overhead of traversing through the regions and column families and their files, if we rely on the filenames to fetch the corresponding metadata.
However, what we found is that path may or may not be always populated by the callers who instantiate BlockCacheKey.
Hence, this change enforces the users and also unit tests to always instantiate/create BlockCacheKey using the paths.
Hence, this change turned out to be a big change.
An alternative to this is another approach that relies only on the file name which is always present in the BlockCacheKey. With this approach, we do not make any changes in the callers of BlockcacheKey or the unit tests.
During cacheEvictions (freeSpace), we will require one traversal through all the files. I had tried to implement this approach in another change: https://github.com/apache/hbase/pull/5829
Please take a look and let me know your idea about the same.

Closing this PR in favour of PR #5829


Add skip_db_file_name option

This will skip the default generation of db.go.
This can be useful when you want to define your own db.go

@yevgenypats This feature can be implemented by wrapping the command that runs sqlc.
sqlc generate
cp ./path/to/other/db.go ./path/to/generated/db.go

Since the work around is simple, I won't be merging this feature in.


21.3.5 -- New scripts now position cursor to the end of script BUG?

System information:

Operating system (distribution) and version : Windows 11 64 Bits
DBeaver version : DBeaver 21.3.5
Additional extensions: None

Connection specification:

Database name and version
Driver name
Do you use tunnels or proxies (SSH, SOCKS, etc)?

Describe the problem you're observing:
Steps to reproduce, if exist:
Include any warning/errors/backtraces from the logs

Hello there,
When I UPDATE my current script, not a new one, the current one, and save the script because I have made some changes, it goes to the end of the script. So I lost my current file position!
It is very unconfortable, I generaly use big scripts... so it is not nice, after saving any change, have to look for the old position to run the script or made another correction.
If you want to make this new change or feature, please make it an option, I want to disable it, but I havent found where!
Thanks for the help and correction!

I'm experiencing the same. A (slightly painful) workaround is to use the "Previous edit location" action after each save, which moves the cursor back to your last change.

Previous edit location

Thank you very much, it is a great idea, I don´t know about this feature, Control-Q save my day...
I hope this bug will be fixed in the next realease.
Thanks a lot!

Hello @luiscarlosbolanhos @nicchristeller,
The issue was fixed in #15562 and will be available in the next version (22.0.0).


Support flow dynamic control for matrix-free

Description of the problem
Flow dynamic control was not being used in the matrix-free operators, which was preventing us from using it to simulate cases like the periodic hills.
Description of the solution
The operators now consider both the source term function and the beta term; the latter only if dynamic flow control is enabled.
How Has This Been Tested?

The results for the first couple of iterations of the periodic hills case on my machine are the same as in the matrix-based solver.
A simple case analogous to the one available for the MB application was included: applications_tests/lethe-fluid-matrix-free/poiseuille3d_flow_control.prm.

Comments
Now we can simulate the periodic hills with lethe-fluid-matrix-free :)

Will merge after CI :)


SIGSEGV has occured (Unbox v1.4)

Hello.
We use Unbox version 1.4, and sometimes SIGSEGV has occured in Unbox processing.
If you have information about similar problem or bug fixes, please teach me.
Stacktrace
Exception Type:  SIGSEGV
Exception Codes: SEGV_ACCERR at 0x20
Crashed Thread:  0

Thread 0 Crashed:
0   libswiftCore.dylib                  0x000000010178988c swift_unknownRetain + 24
1   Unbox                               0x0000000100fc3278 _TTSg5SSSSs8Hashables_Ps9AnyObject____TFVs24_NativeDictionaryStorage8maybeGetfxGSqq__ + 384
2   Unbox                               0x0000000100fbd6c0 _TFC5UnboxP33_B0722540C844FAC3B76F6867342B9C3B18UnboxValueResolver26resolveOptionalValueForKeyurfTSS9isKeyPathSb9transformFxGSqqd____GSqqd___ + 1856
3   Unbox                               0x0000000100fb8d6c _TFC5Unbox7Unboxer5unboxuRxS_16UnboxableRawTyperfTSS9isKeyPathSb_GSqx_ + 208
4   **********                          0x000000010024517c _mh_execute_header + 1675644
5   **********                          0x000000010023d518 _mh_execute_header + 1643800
6   Unbox                               0x0000000100fb5a20 _TF5Unbox12UnboxOrThrowuRxS_9UnboxablerFzTGVs10DictionarySSPs9AnyObject__7contextGSqP___x + 252
7   Unbox                               0x0000000100fb58b0 _TF5Unbox5UnboxuRxS_9UnboxablerFTGVs10DictionarySSPs9AnyObject__7contextGSqP___GSqx_ + 100
8   Unbox                               0x0000000100fc0fa8 _TFFC5Unbox7Unboxer5unboxuRxS_9UnboxablerFTSS9isKeyPathSb_GSqx_U_FGVs10DictionarySSPs9AnyObject__GSqQ__ + 160
9   Unbox                               0x0000000100fbd850 _TFC5UnboxP33_B0722540C844FAC3B76F6867342B9C3B18UnboxValueResolver26resolveOptionalValueForKeyurfTSS9isKeyPathSb9transformFxGSqqd____GSqqd___ + 2256
10  Unbox                               0x0000000100fbaf98 _TFC5Unbox7Unboxer5unboxuRxS_9UnboxablerfTSS9isKeyPathSb_GSqx_ + 224

iOS version
8.1.2, 10.3.1, 10.3.2.
How we use Unbox
We use Unbox with Unboxer Struct with properties of String and nested Unboxer Struct.
This problem has occured in our customer's environment, and it has't occured in our development environment. So i'm sorry about few information.

Hi @jn000, can you upgrade to the latest version of Unbox (2.5.0)? The version you're using is almost 1,5 years old, so I'm unable to provide support for it.

@JohnSundell
Thank you for your immediate reply.
We got it and consider your suggestion.


drivers: nrf: Convert nrf drivers to new DT_INST macros

Convert older DT_INST_ macro use in nrf drivers to the new
include/devicetree.h DT_INST macro APIs.
Signed-off-by: Kumar Gala<EMAIL_ADDRESS>

I can handle getting the entire soc directory moved to the new DT API if you'd prefer.

@galak I pushed https://github.com/mbolivar-nordic/zephyr/tree/validate-addr-dt to rework the validate_base_addresses.c part. Depends on https://github.com/zephyrproject-rtos/zephyr/pull/23753.
The rest seems fine to me.

@galak and @mbolivar I think this is stuck waiting for an update after @anangl's comments.

@galak and @mbolivar I think this is stuck waiting for an update after @anangl's comments.

This is waiting on me to convert our driver over wholesale. I discussed with @galak and our plan is to convert individual drivers over to the new API in separate PRs. Will close this for now.


Feature request: Porting to OpenBSD

I  tried to compiled from source on OpenBSD 7.5 and I got this errors
Using go build -o wakapi
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x17e1308]

goroutine 1 [running]:
modernc.org/libc.(*TLS).setErrno(0x2532f80?, {0x2a8ea0?, 0x72ff00?})
        /home/user/go/pkg/mod/modernc.org/libc@v1.49.0/etc.go:192 +0xa8
modernc.org/libc.Xmalloc(0x0, 0x340320?)
        /home/user/go/pkg/mod/modernc.org/libc@v1.49.0/mem.go:39 +0xc5
modernc.org/libc.init()
        /home/user/go/pkg/mod/modernc.org/libc@v1.49.0/libc_openbsd.go:49 +0x3b6

or go install
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x178b6c8]

goroutine 1 [running]:
modernc.org/libc.(*TLS).setErrno(0x24dd380?, {0x2a8ea0?, 0x6d81a0?})
        /home/user/go/pkg/mod/modernc.org/libc@v1.49.0/etc.go:192 +0xa8
modernc.org/libc.Xmalloc(0x0, 0x340320?)
        /home/user/go/pkg/mod/modernc.org/libc@v1.49.0/mem.go:39 +0xc5
modernc.org/libc.init()
        /home/user/go/pkg/mod/modernc.org/libc@v1.49.0/libc_openbsd.go:49 +0x3b6

Any hint on what should I do or can be done to port it on OpenBSD?

Hi @samarulmeu, thanks for reporting this. I've never worked with BSD and have little knowledge about how it differs from Linux, so can't really help a lot on this end. There seems to be an issue with the Go port of libc for OpenBSD, see cznic/libc#34. There's also a discussion around OpenBSD support in the upstream repo. Perhaps you can find some useful insights there?
Even though this probably isn't anything that could be fixed on our end, I'd love to stay in the loop. Keep us posted if you get Wakapi to work on OpenBSD :nerd_face:

So this is my update as I finally managed to get it working on OpenBSD.
These are the steps I followed. (Take into account I am not a go specialist so what I write might be wrong ;) )
First you need to upgrade go libraries with go get -u (I tested on 7.5, it will be interesting to see if you need this upgrade in current)
After that, when you try to do a go build -o wakapi you get this error
# github.com/muety/wakapi/services/mail
services/mail/smtp.go:46:14: c.StartTLS undefined (type *smtp.Client has no field or method StartTLS)

What I did was to comment the lines 46-53 from services/mail/smtp.go compiled and I got the executable and the server run. I'm sure there will be no mail functionality but for running locally will be ok I think. The problem might be in  the OpenBSD mail implementation.
So, we might still dig further or/and add a installation instruction for OpenBSD users.

Thanks for sharing your insights! The error is unrelated to OpenBSD and goes back to breaking changes in an upstream library, see #657. So in essence, it seems like running Wakapi on OpenBSD is nothing more but a matter of upgrading to latest dependency versions. I'll push an updated version soon!


collection idea: "napari plugins that work well with OME-Zarr"

cc @jluethi @dgmccart

We are archiving this repository as a part of the deprecation of collections on napari-hub. We will not continue to work on this repository. Therefore all issues and pull requests are being closed. Thanks for your contribution.


Fix vultr-snapshot-id flag being ignored

Description
This PR changes the flag parsing to properly pass the given snapshot ID to the API request.
Related Issues

Checklist:

[x] Have you checked to ensure there aren't other open Pull Requests for the same update/change?
[x] Have you linted your code locally prior to submission?
[x] Have you successfully ran tests with your changes locally?

This was fixed in #25

just in time lol

@cienijr Test it out and let us know if you have any issues.


Final Milestone: Preserve data in the browser

Portfolio project's Final milestone: Preserve data in the browser's local storage
In this milestone, we:

Implemented the following interactions in the form:

When the user changes the content of any input field, the data is saved to the local storage.
When the user loads the page, if there is any data in the local storage the input fields are pre-filled with this data.

Used the following data model:

Created a single JavaScript object with all the data from the entire form and save it in local storage.

Updated Future features in README.

General requirements checklist.

[X] No linters error.
[X] Followed the GitHub flow.
[X] documented work professionally.
[X] Followed the HTML, CSS and JavaScript best practices.

Group PR for this milestone


Make reminder expire if not scheduled

Closes #228
Closes #207

Codecov Report

Merging #231 into master will increase coverage by 0.88%.
The diff coverage is 78.37%.

@@             Coverage Diff              @@
##             master     #231      +/-   ##
============================================
+ Coverage     69.81%   70.69%   +0.88%
- Complexity      975      998      +23
============================================
  Files           146      147       +1
  Lines          3174     3225      +51
  Branches        348      352       +4
============================================
+ Hits           2216     2280      +64
+ Misses          834      817      -17
- Partials        124      128       +4

Impacted Files
Coverage Δ
Complexity Δ

.../seedu/momentum/logic/commands/AddTaskCommand.java
94.11% <ø> (ø)
7.00 <0.00> (ø)

...c/main/java/seedu/momentum/model/ModelManager.java
71.36% <0.00%> (-0.20%)
60.00 <0.00> (+1.00)
:arrow_down:

...java/seedu/momentum/model/project/TrackedItem.java
93.67% <ø> (+1.26%)
29.00 <0.00> (+1.00)

...c/main/java/seedu/momentum/ui/TrackedItemCard.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...ava/seedu/momentum/commons/core/ThreadWrapper.java
62.50% <62.50%> (ø)
3.00 <3.00> (?)

...seedu/momentum/model/reminder/ReminderManager.java
97.26% <84.61%> (+13.67%)
20.00 <2.00> (+1.00)

...n/java/seedu/momentum/model/reminder/Reminder.java
92.68% <88.00%> (-7.32%)
28.00 <16.00> (+10.00)
:arrow_down:

...rc/main/java/seedu/momentum/model/ProjectBook.java
94.00% <100.00%> (+20.19%)
23.00 <4.00> (+4.00)

...ain/java/seedu/momentum/model/project/Project.java
88.33% <100.00%> (+7.20%)
22.00 <2.00> (+3.00)

...c/main/java/seedu/momentum/model/project/Task.java
90.90% <100.00%> (+10.90%)
10.00 <1.00> (+2.00)

... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2787041...7af1005. Read the comment docs.


Add LNPopupController, LNInterpolation and LNChevronView to repo.

Project URL
https://github.com/LeoNatan/LNPopupController
https://github.com/LeoNatan/LNInterpolation
https://github.com/LeoNatan/LNChevronView
Description
Adding three of my frameworks. LNPopupController is already popular. The other two are new.
Checklist

[ ] Only one project/change is in this pull request
[x] Addition in chronological order (bottom of category)
[x] Supports iOS 8 or later
[x] Supports Swift 3
[x] Has a commit from less than 2 years ago
[x] Has a clear README in English

I added three frameworks, and added them in one PR. 😃 They are all mine, so there should be no issues.

          1 Error

  :no_entry_sign:
  Found 1 link issue

          1 Warning

  :warning:
  Too many changes (when adding, please keep it to one project per Pull Request)

Link issue by awesome_bot

Line
Status
Link

1445
301
https://github.com/OEASLAN/OEANotification redirects tohttps://github.com/OEA/OEANotification

  Generated by :no_entry_sign: danger

README.md in both Swift and Objective-C? Love it! 🎉🎉🎉🎉

The only problem i found is in LNChevronView.
We encourage people to send a pull request for each library to avoid problems like this, on where one of the libraries is not valid, but the other ones are

Please fix this issue

Also this one

Can you tell me what is invalid so I can fix? The chevron view is a very simple framework.

@LeoNatan just opened issues. Let me know what you think 😉

I do not support CocoaPods. I'll add Carthage instructions soon.

@LeoNatan CocoaPods is not a requirement for libs in AwesomeiOS, but we request an installation section on the Readme.md

Cool. It's good to have, I agree.

@lfarah I updated the repositories with installation instructions.

@LeoNatan can you add more information on LNChevronView? You can check a complete one HERE

@LeoNatan let me know if you need any help!

@LeoNatan I think it would be cool to say where this chevron view is used, how to implement on Storyboard and how to implement programmatically. There are a lot of people new to programming and they require more information.

OK, I'll add some more information later.

@lfarah Added more information to the README.

@LeoNatan perfect! Now you just have to rebase because there are merge conflicts.
Lemme know if you need any help! 👍

@lfarah Merged.

Bad merge, should not be deleting anything
Can u fix it

@dkhamsing  Ehh, I wonder what happened. Will take care of that soon.

closing for now, feel free to submit again
we prefer you send one project per pull request
thanks!


Make TranslatorInterface compatible with Symfony 6

Fixes https://github.com/Behat/Behat/issues/1363

Thanks for the PR, David! It looks like it's fixed in Symfony 6 already so I'll close it.

@pamil I don't follow. How was this fixed on Symfony's side exactly? The issue still happens for me


Failed to build apiserver due to bazel errors

I am trying build my own apiserver step by step instructed by :https://github.com/kubernetes-incubator/apiserver-builder-alpha/commits/example-simple
But I ran into unexpected error in step 6, it tells me that I shoud execute  apiserver-boot build executables --bazel --gazelle --generate=false
And it ends with errors like:
bazel run //:gazelle
Starting local Bazel server and connecting to it...

ERROR: /home/wlh/mywork/go/packages/src/github.com/kubernetes-incubator/apiserver-builder-alpha/WORKSPACE:2:1: name 'http_archive' is not defined
ERROR: Error evaluating WORKSPACE file
ERROR: error loading package '': Encountered error while reading extension file 'go/def.bzl': no such package '@io_bazel_rules_go//go': error loading package 'external': Could not load //external package
ERROR: error loading package '': Encountered error while reading extension file 'go/def.bzl': no such package '@io_bazel_rules_go//go': error loading package 'external': Could not load //external package
INFO: Elapsed time: 1.413s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)
2019/05/17 18:21:56 exit status 1

Could anyone help

@pwittrock Any thoughts?

hi @WanLinghao, thanks for reporting this. it's sort of a duplication of #312. bazel build is broken for now for some issue.  please remove the options --bazel --gazelle and run again the command. and i'll the revise the guide in a follow-up.