1/40th config with existing yr156 ACCESS-OM2-01 forcing?

The Sapphire Rapid nodes are now online, so there is a bit of time pressure to get things going. (Though I have yet to hear from Ben or Al that they're ready for us.)
We had hoped to use year 2 boundary forcing (either OM4 or ACCESS-OM2-01), so that we have a boundary condition closer to obs with less bias. However we're struggling to get that working.
Should we consider an alternative option using the existing yr 156 forcing on the 1/40th config, so at least we can run something? I can no longer find the raw ACCESS-OM2-01 yr 156 data, which I think may have been deleted. However we could interpolate the existing 1/20th forcing (/g/data/x77/ahg157/inputs/mom6/panan/forcing_obc_shifted.nc) onto the 1/40th grid.
We know that the run with the yr 156 forcing data has a large drift at the northern boundary, but if our focus is on the Antarctic shelf and slope processes in the upper ocean, perhaps that doesn't matter so much. It would be useful to plot some trend plots of circulation and T/S near the Antarctic slope at a few different depths (perhaps 500m, 1000m, 2000m) for the first 10-20 years of the panant-01-zstar-v13 simulation to see if drift is an issue there.

I think running something is better than nothing. There are no guarantees that the 2 year boundary forcing won't drift even if they worked. Also, we will learn something regardless of the drift by running the simulation (we will probably mess something else up in the simulation anyway). I will aim to check the drift near the slope in the first 10-20 years of the panant-01-zstar-v13 simulation.

Awesome thanks Paul! I have also just started a 1/20th case with the new
topography and old yr 156 ACCESS-OM2-01 initial conditions and boundary
forcing. Once that runs out a way we can also check the drift in that near
the slope.
On Tue, 14 Mar 2023 at 14:32, Paul Spence @.***> wrote:

I think running something is better than nothing. There are no guarantees
that the 2 year boundary forcing won't drift even if they worked. Also, we
will learn something regardless of the drift by running the simulation (we
will probably mess something else up in the simulation anyway). I will aim
to check the drift near the slope in the first 10-20 years of the
panant-01-zstar-v13 simulation.
—
Reply to this email directly, view it on GitHub
https://github.com/COSIMA/mom6-panan/issues/29#issuecomment-1467298790,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACA44UYX7NGAB6OEDQVHRHLW37RFRANCNFSM6AAAAAAVZZGOQQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

@willaguiar has most of the 1/40th files ready to go (bathymetry, masks, runoff) and will work on interpolating the boundary forcing (/g/data/x77/ahg157/inputs/mom6/panan/forcing_obc_shifted.nc) and initial conditions (/g/data/x77/ahg157/inputs/mom6/panan/temp_salt_init_z_0025_filled_NaNs.nc) from the 1/20th config.
The one thing we're not sure about is how we decide on the tile layout and generate the mask_table file for the 1/40th. @micaeljtoliveira @angus-g any chance that one of you could jump onto this for us?

The one thing we're not sure about is how we decide on the tile layout and generate the mask_table file for the 1/40th.

Would this be the processor layout and mask?

Would this be the processor layout and mask?

Yes, that's it @micaeljtoliveira.

I can have a go at it. Can you just point out where I can find the grid and topography files?

Thanks @micaeljtoliveira! It's this config. Currently it's very messy pointing to multiple directories with identically named (but different data) files! :grimacing:
From what I can tell it is using /g/data/e14/cs6673/panan-005/domain-tools/new_topog/ topog.nc and /g/data/e14/cs6673/panan-005/domain-tools/new_topog/ ocean_hgrid_0025.nc.

@adele157 Sorry,  I'm a bit confused here. The files /g/data/e14/cs6673/panan-005/domain-tools/new_topog/topog.nc and /g/data/e14/cs6673/panan-005/domain-tools/new_topog/ocean_hgrid_0025.nc are for a 1/20deg resolution, but I would need the 1/40deg files. Maybe @willaguiar could point me in the right direction?

Ooops, sorry, brain was clearly not working!
I think they are these ones (@willaguiar can confirm these are the latest versions):
/scratch/g40/wf4500/mom6_topo/shared/ocean_hgrid37S_2.nc
/scratch/g40/wf4500/mom6_topo/shared/topog_37S.nc

Ooops, sorry, brain was clearly not working!
I think they are these ones (@willaguiar can confirm these are the latest versions): /scratch/g40/wf4500/mom6_topo/shared/ocean_hgrid37S_2.nc /scratch/g40/wf4500/mom6_topo/shared/topog_37S.nc

Yes these should be the files for the 1/40th. ocean_mask.nc should be in the same directory too.

@willaguiar To generate the computational mask, I need a ocean mosaic file. I generated it using the make_solo_mosaic utility, but it seems there's some inconsistency in the grid. When running check_mask I get the following error:
Error from read_mosaic: mismatch between refine_ratio and istart_in/iend_in
I also tried to generate the exchange grid, but got another error, probably related:
from read_mosaic_grid_sizes: ny is not divided by y_refine
Did I miss something?

@willaguiar To generate the computational mask, I need a ocean mosaic file. I generated it using the make_solo_mosaic utility, but it seems there's some inconsistency in the grid. When running check_mask I get the following error:
Error from read_mosaic: mismatch between refine_ratio and istart_in/iend_in
I also tried to generate the exchange grid, but got another error, probably related:
from read_mosaic_grid_sizes: ny is not divided by y_refine
Did I miss something?

@micaeljtoliveira I just transferred to the folder the mosaic I previously created:
/scratch/g40/wf4500/mom6_topo/shared/ocean_mosaic.nc
Let me know in case you need the land and atmospheric mosaic too.

@micaeljtoliveira Some information that probably is useful:
To create the ocean_mask.nc I used the ocean_hgrid37S.nc instead of ocean_hgrid37S_2.nc.  ocean_hgrid37S.nc has ny=nyp-2, and  ocean_hgrid37S_2.nc has ny=nyp-1.
For some reason having ny=nyp-1 in ocean_hgrid37S_2.nc would generate the same errors you found when using make_solo_mosaic and check_mask. But both tools worked fine in a grid with ny=nyp-2 (i.e., ocean_hgrid37S.nc).

Update: the Sapphire Rapids are being publicly released next Wednesday! @willaguiar, @fabiobdias or others, it would be awesome if you have any time available to try to make this 1/40th config happen!

That's strange. I suspect that you cut the grid in the wrong place and were left with half cells. Normally nyp must be odd and ny should always be nyp-1. @angus-g Could you confirm this?

Normally nyp must be odd and ny should always be nyp-1. @angus-g Could you confirm this?

I don't think that nyp must be odd, it just defines the number of cell edges. So a single-cell grid would have nyp=2 And ny defines the number of cell centres which should indeed be nyp-1.

@angus-g @micaeljtoliveira I knew that ny should be = nyp-1, but didn't realized nyp should be odd (makes sense). That is probably why I was having the same error as  @micaeljtoliveira.
I remade the grid and masks, (now with nyp odd) and updated the files  now.
List of files:
ocean grid: /scratch/g40/wf4500/mom6_topo/shared/ocean_hgrid_panant0025.nc
ocean topography: /scratch/g40/wf4500/mom6_topo/shared/topog_panant0025.nc
ocean mask: /scratch/g40/wf4500/mom6_topo/shared/ocean_mask_panant0025.nc

@willaguiar Thanks for fixing the issue with the grid!
I've now created three computational masks:

mask_table.580.100x25
mask_table.596.102x25
mask_table.1198.141x35

The first one uses 1920 cores and is suitable to use on Cascade Lake nodes. The second one uses 1954 cores and meant to be used with the Sapphire Rapids nodes. The last one uses roughly twice the number of cores of the previous ones (3737) and should be suitable for both Sapphire Rapids and Cascade Lake nodes.
I've also generated the mosaic and exchange grids. Have gathered all the files, including grid, topography and ocean mask in one place: /g/data/ik11/inputs/mom6/panan/0025deg/20230320

Good news! The rerun panan-01 cases with yr2 ACCESS-OM2-01 or OM4 boundary forcing are looking good! They've only run a year so far, so I'll run out a bit more. But I think it's looking like we'll be able to switch the 1/20th and 1/40th to the new forcing. I guess we can see how these look after a few more years and decide whether we want OM4 or ACCESS-OM2-01 forcing.

Hello everyone. I finished interpolating the boundary forcing (BF). The directory for the boundary forcing I created and for the initial conditions by @fabiobdias are listed in the end of the post. The BF for the 1/40th was created by interpolating the panant1/20th BF to the new grid.
I made some simple plots to compare the 1/20th and 1/40th BF here and they seem equal to me. But I need someone to confirm if the variables "ilist_segment_0001" and "jlist_segment_001" are correct. "ilist" seem to be a list of longitude indexes, but the "jlist" in the source file was a list of repeated values =1942 (so I'm not sure what it really means).
files:
Interpolated boundary forcing for 1/40th: /g/data/x77/wf4500/forcing_final_140th.nc
Interpolated initial conditions for 1/40th: /scratch/v45/fbd581/mom6-panan/scripts/temp_salt_init_z_panant0025.nc

@willaguiar Thanks for fixing the issue with the grid!
I've now created three computational masks:

mask_table.580.100x25
mask_table.596.102x25
mask_table.1198.141x35

The first one uses 1920 cores and is suitable to use on Cascade Lake nodes. The second one uses 1954 cores and meant to be used with the Sapphire Rapids nodes. The last one uses roughly twice the number of cores of the previous ones (3737) and should be suitable for both Sapphire Rapids and Cascade Lake nodes.
I've also generated the mosaic and exchange grids. I have gathered all the files, including grid, topography and ocean mask in one place: /g/data/ik11/inputs/mom6/panan/0025deg/20230320

I tried to run Panan-0025 but the run is killed with the following error on mom6.out
68 total processes killed (some possibly by mpirun during cleanup)
and on error log
Model exited with error code 137; aborting.
@micaeljtoliveira Could that be related to the computational masks?

What's the path of your run directory? Might be helpful to see how far it
got through initialisation or if there are any other error messages.
On Wed, 29 Mar 2023 at 18:15, Wilton Aguiar @.***>
wrote:

@willaguiar https://github.com/willaguiar Thanks for fixing the issue
with the grid!
I've now created three computational masks:

mask_table.580.100x25
mask_table.596.102x25
mask_table.1198.141x35

The first one uses 1920 cores and is suitable to use on Cascade Lake
nodes. The second one uses 1954 cores and meant to be used with the
Sapphire Rapids nodes. The last one uses roughly twice the number of cores
of the previous ones (3737) and should be suitable for both Sapphire Rapids
and Cascade Lake nodes.
I've also generated the mosaic and exchange grids. I have gathered all the
files, including grid, topography and ocean mask in one place:
/g/data/ik11/inputs/mom6/panan/0025deg/20230320
I tried to run Panan-0025 but the run is killed with the following error
on mom6.out
68 total processes killed (some possibly by mpirun during cleanup)
and on error log
Model exited with error code 137; aborting.
@micaeljtoliveira https://github.com/micaeljtoliveira Could that be
related to the computational masks?
—
Reply to this email directly, view it on GitHub
https://github.com/COSIMA/mom6-panan/issues/29#issuecomment-1488065475,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACA44U6J6OMWVNY7TBDCX2DW6POSRANCNFSM6AAAAAAVZZGOQQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

It crashes right when it initializes. mom6.err/out seem to indicate a problem with y in the hgrid
dir: /home/156/wf4500/mom6-panan
mom6.out:
NOTE from PE     0: grid_mod/get_grid_cell_vertices: domain is not present, global data will be read
NOTE from PE     0: MPP_IO_SET_STACK_SIZE: stack size set to  194896367.
NOTE from PE     0: grid_mod/get_grid_cell_centers: domain is not present, global data will be read

mom5.err
FATAL from PE   504: NETCDF ERROR: NetCDF: HDF error File=INPUT/ocean_hgrid.nc Field=y

It crashes right when it initializes. mom6.err/out seem to indicate a problem with y in the hgrid dir: /scratch/g40/wf4500/mom6/
mom6.out:
NOTE from PE     0: grid_mod/get_grid_cell_vertices: domain is not present, global data will be read
NOTE from PE     0: MPP_IO_SET_STACK_SIZE: stack size set to  194896367.
NOTE from PE     0: grid_mod/get_grid_cell_centers: domain is not present, global data will be read

mom5.err
FATAL from PE 504: NETCDF ERROR: NetCDF: HDF error File=INPUT/ocean_hgrid.nc Field=y

It seems that a similar grid error with MOM5 happened (at least on mom.out) here.
Does anybody know what was the solution for that one?

@willaguiar I think you forgot to update NIGLOBAL and NJGLOBAL in the MOM_input and SIS_input files. I believe these should be:
NIGLOBAL = 14400
NJGLOBAL = 3383

This seems to work, but then the run crashes because of not enough memory. I then switched to using 3737 cores on the new Sapphire Rapids nodes, but still crashed by lack of memory.
I will try to create a new mask using a larger number of cores and see how that goes.

@willaguiar I think you forgot to update NIGLOBAL and NJGLOBAL in the MOM_input and SIS_input files. I believe these should be:
NIGLOBAL = 14400
NJGLOBAL = 3383

This seems to work, but then the run crashes because of not enough memory. I then switched to using 3737 cores on the new Sapphire Rapids nodes, but still crashed by lack of memory.

Yeppp. I updated N*GLOBAL in other tries and it gave the same error. We just realized it was memory problem too.

I will try to create a new mask using a larger number of cores and see how that goes.

thanks!

We did discover that if you don't explicitly set the layout for the (null) atmos/land models, you can get some pretty bad layouts! Using 1954 cores, that's only divisible by 2 and 977, so you get a lot of extremely thin tiles in those components. They're not doing anything significant, but it probably harms IO through the coupler... I think maybe we should set the layout (and mask) to match the ocean/ice models: in input.nml:
&atmos_model_nml
    layout = X,Y
    mask_table = 'INPUT/mask_table_name'
/

&land_model_nml
    layout = X,Y
    mask_table ='INPUT/mask_table_name'
/

I just tried it with 7563 cores, which corresponds to 13.69TB of memory, but it still did not work. Need more cores... Unfortunately I won't be able to try this until tonight.
In the meantime, is there a way to roughly estimate the total memory needed by a MOM6 calculation based on the number of grid points?

It seems I've got something that runs! It still crashes, but it seems that's because I don't have permissions to read /home/156/wf4500/mom6-panan/input_1_40th/ . The main issue was my lack of experience with payu. When switching to the Sapphire Rapid nodes I forgot to explicitly add the amount of memory to request. These nodes have 512Gb memory each, so we there was a lot more available than what payu was requesting.
In any case, I generated a new mask, as we will definitely need to use more cores so that this runs fast enough. I also added the layout and mask for the atmos/land models. I also set the IO_LAYOUT to a more reasonable value. My files are here: /scratch/g40/mo1833/mom6/work/0.025

Thanks so much Micael! I opened permissions for the '/home/156/wf4500/mom6-panan/input_1_40th/' now. Imma try to run it in my dir too.

update. Memory, mask problem is gone now with the new mask_table.2431.204x48 file and 7361 cpus.
panan0025 runs, but it crashes with the following error:
FATAL from PE  7280: Invalid data type for dimension
It doesn't point to a specific file tho. Can someone tell me what comes after those lines below on panan005 runs? It might help me identify which input file has invalid data type.
 Memory, mask problem is gone now with the new
mask_table.2431.204x48 file and 7361 cpus.
panan0025 runs, but it crashes with the following error:
FATAL from PE 7280: Invalid data type for dimension
It doesn't point to a specific file tho. Can someone tell me what comes
after those lines below on panan005 runs? It might help me identify
which input file has invalid data type.

You are receiving this because you were mentioned.Message ID:
@.***>

Update and good news!
Problem above was caused because the boundary forcing had dimensions in int64 instead of int. Other errors appeared, but were fixed by setting time as record dimension in the boundary forcing, and adding coordinates as data in the initial conditions file.
Good news is .... I was able to run panan0025 for 1 model day without any error messages!
Caveat: memory was still too low, and the run was killed after 1 model day because of lack of memory (~30TB). I will request a higher memory and rerun it.
new BF: /g/data/x77/wf4500/mom6-panan-1_40th/forcing_final_140th_int3.nc
new IC: /scratch/v45/fbd581/mom6-panan/scripts/temp_salt_init_z_panant0025_31032023.nc
perhaps someone with writing permissions to  /g/data/ik11/inputs/mom6/panan/0025deg/20230320 should add these files to that folder, since they are working fine now.
On another note.... Since we have the model running now, should we already change the bathymetry as in here, before doing another run?

Great that it's running! Yes, let's update the bathymetry. I can do it so it's the same as the 1/20th that I just did.

Update: I have edited the depth variable in the topography files for the 1/20th and 1/40th here:
/g/data/v45/akm157/inputs/mom6/panan/panan_0025/topog_edited.nc and
/g/data/v45/akm157/inputs/mom6/panan/panan_005/topog_edited.nc
But... I haven't changed the sea_area_fraction variable, as I'm not sure how that's supposed to be set. And I haven't redone any of the masks.

On the mask subject... I rerun panan0025 using mask_table3093.220x56 (9227 ncpus, 46Tb) and the run still crashed within 1 model day due to lack of memory.  (Perhaps we will need a mask that supports more cpus).

On the mask subject... I rerun panan0025 using mask_table3093.220x56 (9227 ncpus, 46Tb) and the run still crashed within 1 model day due to lack of memory. (Perhaps we will need a mask that supports more cpus).

I had a look at the scaling tests I did with the lower resolution configurations, and it seems that the memory used increases with the number of cores almost linearly. Although some increase is to be expected, a linear increase means that there's a some large data duplicated among cores, which in turn means that just increasing the number of cores might not solve out problem.
The fact that the problem only shows up after 1 model day might indicate that it happens during I/O. I'm assuming you are doing some daily output?
@angus-g Any thoughts on this?

On the mask subject... I rerun panan0025 using mask_table3093.220x56 (9227 ncpus, 46Tb) and the run still crashed within 1 model day due to lack of memory. (Perhaps we will need a mask that supports more cpus).

I had a look at the scaling tests I did with the lower resolution configurations, and it seems that the memory used increases with the number of cores almost linearly. Although some increase is to be expected, a linear increase means that there's a some large data duplicated among cores, which in turn means that just increasing the number of cores might not solve out problem.

Hi @micaeljtoliveira @angus-g
I was looking digging into the error logs today and found the crash doesn't happen due to memory -  sorry. I thought that before because the total memory used was ~40Tb close to the total requested, but it turns out the crash wasn't because of it).

The fact that the problem only shows up after 1 model day might indicate that it happens during I/O. I'm assuming you are doing some daily output?

Yes, I was running a 3-days test just to see if the model ran.
The actual model error seem to be due to a mismatch between the BF dates and the run dates.
FATAL from PE  9055: time_interp_external 2: time 726350 (19910101.001500 is after range of list<PHONE_NUMBER>99(19000101.120000 - 19001231.120000),file=INPUT/forcing_final_140th_int3.nc,field=u_segment_001
The weird thing tho is that on the forcing file for the 1/20th the time is exactly the same, and so is the current_date on input.nml (current_date = 1991, 1, 1, 0, 0, 0) between the runs. I will alter the date in the BF file to see if the problem is fixed tho.

@willaguiar You need to set themodulo attribute on the forcing, something like:
ncatted -O -a "modulo,time,a,c " forcing.nc

@angus-g I've set the IO layout to 51,12, for a core layout of 204,48, as it seemed that doing the I/O on a single core was not really a reasonable option anymore with such a high resolution. Not sure if this layout is a good choice though. At least it looked reasonable to me...

I think the ideal would be for the IO layout to line up with the nodes (i.e. one IO core per node), but I guess FMS doesn't have any notion of the physical layout of the cores. Sounds reasonable to me too!

Panan 1/40th is now running smooth with the new topography, and initial conditions and bathymetry for year 2 of OM2-01. Running 2 model days with 9227 CPUs takes 23 minutes walltime ( not counting initialization and checksum). So we will need more than 5hrs to run a full month.
@micaeljtoliveira   , do you think it is worth increasing/doubling the cpus so we can have monthly outputs? Alternatively we can ask for an exception to run the model for more than 5hrs

@willaguiar I can try to find the best number of cores to get a full month done under 5 hours. I was anyway planning to do a scaling test as soon as we had something working, so it's good to have a clear target for it. Probably also worth tuning the I/O layout.

@willaguiar Could you create a new branch for your 1/40th degree configuration and push it, so that I can use it for my tests?

@micaeljtoliveira just done it! Branch named panan-0025

FYI, I’ve also asked NCI if we can have a longer wall time limit for the
1/40th runs, and I think they will approve (have asked for 10 hrs). Though
it would still be good to run on more cores if possible to get a bit faster
throughput, if the scaling is not too terrible.
On Wed, Apr 12, 2023 at 11:39 AM, Wilton Aguiar @.***>
wrote:

@micaeljtoliveira https://github.com/micaeljtoliveira just done it.
Branch named panan-0025
—
Reply to this email directly, view it on GitHub
https://github.com/COSIMA/mom6-panan/issues/29#issuecomment-1504401646,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACA44U5D6LG54BTMIMON5KLXAYBVBANCNFSM6AAAAAAVZZGOQQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

Looks like the maximum number of cores one can use with the Sapphire Rapids nodes is 10400, so I'm now trying to find out what is the maximum number of cores we can use with our configuration within that limit. I've got a reasonable candidate (10383 cores), but I need to check that it runs fine.

10400 seems very low - that's less than the 12144 we were using on cascade lake. What sets this limit?

It's a queue limit: https://opus.nci.org.au/display/Help/Queue+Limits

There are about half as many sapphire rapid cores as cascade lake (74880 vs 147552  https://nci.org.au/our-systems/hpc-systems) so I guess that's why the halved the maximum core count. But 10400 is only ~14% of the cores so they could increase the limit if they wanted.
@adele157 can you ask your contact at NCI why here's such a low core count limit on sapphire rapids? The limit they've set (10400) will make it impossible for us to run our normal production configuration for ACCESS-OM2-01-BGC (12144 cores) on the new nodes.

Using 10383 cores seems to works fine (my run reached the main loop) but I didn't get any timings. It would also be good to optimize the IO layout. Unfortunately these tests are using a lot of SU's (> 30 KSU), so it's better to discuss the next steps during the weekly meeting.

I’m guessing you’ve already discussed this today, but I’m case it’s useful:
We have enough hours for testing Micael, go for it (ol01 or elsewhere).
I can ask if they can increase our SU limit too, what number would we want?
On Mon, Apr 17, 2023 at 10:54 AM, Micael Oliveira @.***>
wrote:

Using 10383 cores seems to works fine (my run reached the main loop) but I
didn't get any timings. It would also be good to optimize the IO layout.
Unfortunately these tests are using a lot of SU's (> 30 KSU), so it's
better to discuss the next steps during the weekly meeting.
—
Reply to this email directly, view it on GitHub
https://github.com/COSIMA/mom6-panan/issues/29#issuecomment-1510546442,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACA44UZHKW56ZCZNWRT2ET3XBSICFANCNFSM6AAAAAAVZZGOQQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

@adele157 Thanks!
Looks like we have used almost all the hours in ol01. I seem to remember there was a specific project with hours for the Sapphire Rapids nodes. Is it still the case? Otherwise what's the best project for this? e14? Note that just the testing is likely use a few hundred thousand SU's, as a short run with ~10000 cores uses 30-40 KSU.

Yes, ot56, go for it!
On Mon, Apr 17, 2023 at 3:08 PM, Micael Oliveira @.***>
wrote:

@adele157 https://github.com/adele157 Thanks!
Looks like we have used almost all the hours in ol01. I seem to remember
there was a specific project with hours for the Sapphire Rapids nodes. Is
it still the case? Otherwise what's the best project for this? e14? Note
that just the testing is likely use a few hundred thousand SU's, as a short
run with ~10000 cores uses 30-40 KSU.
—
Reply to this email directly, view it on GitHub
https://github.com/COSIMA/mom6-panan/issues/29#issuecomment-1510709855,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACA44U7IROBUFNKQRE3SERLXBTF53ANCNFSM6AAAAAAVZZGOQQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

I now have the timings when using 10383 cores. Unfortunately it still takes too long: two simulation days take a bit over 20 minutes walltime. This means we definitely need a higher core count or a longer walltime.

Wilton can now run 10 hours on 100 nodes (10400 cores) on normalsr on
project ot56. Probably we need access for Micael also? Is this number of
cores ok, or do we need to increase slightly to accomodate the tests you’ve
been doing on 10383 cores?
On Tue, Apr 18, 2023 at 10:38 AM, Micael Oliveira @.***>
wrote:

I now have the timings when using 10383 cores. Unfortunately it still
takes too long: two simulation days take a bit over 20 minutes walltime.
This means we definitely need a higher core count or a longer walltime.
—
Reply to this email directly, view it on GitHub
https://github.com/COSIMA/mom6-panan/issues/29#issuecomment-1512279600,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACA44U5TO6OLXBNTCHDX4XDXBXPAZANCNFSM6AAAAAAVZZGOQQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

Yes! I responded the NCI ticket asking to add Micael to the walltime exception, and also with a request to increase the ncores. @micaeljtoliveira is still running some tests to find the optimal configuration (which could be slightly above the queue ncores limit), so in the last meeting we decided to ask for an increase in the ncores for these runs.

Wilton can now run 10 hours on 100 nodes (10400 cores) on normalsr on project ot56. Probably we need access for Micael also? Is this number of cores ok, or do we need to increase slightly to accomodate the tests you’ve been doing on 10383 cores?
…

After a few more tests, I got to the conclusion it was best to use slightly less cores (9872) and a different IO layout. It takes almost the same time to run one month (~6h20), but uses less SU's (~125 KSU instead of ~129 KSU).
Unfortunately the initialization and termination still take a large fraction of the total walltime (~15%), so it might be worth trying to do 2 months per run, but for that we need to use more cores than the current limit.
I'll update the panan-0025 branch with the new core count and mask.
@angus-g It looks like the best IO layout is to have roughly one IO core per compute node, as you predicted!

@angus-g It looks like the best IO layout is to have roughly one IO core per compute node, as you predicted!

Glad to know the theory still applies in practice. Thanks for doing these tests! I can look into whether we can get any easy wins for initialisation/termination so it's not such a large chunk of a run segment?

I can look into whether we can get any easy wins for initialisation/termination so it's not such a large chunk of a run segment?
That would be great!

A simple one that I don't know how we missed: PARALLEL_RESTARTFILES will write one restart file per IO core, which at least speeds up the termination significantly. This option can be used for both MOM and SIS, although I forgot to set a non-unity IO layout for SIS, so it only applies to MOM for the numbers here.
Then for the test run I did, of the 70s initialisation time, 26s was in setting up the exchange grid, 15s in initialising SIS, and 13s initialising MOM. It also brought the termination time down to about 30s from 210s.

@angus-g Thanks! I've tried setting PARALLEL_RESTARTFILES and  IO_LAYOUT for both MOM and SIS. In my tests, this brings the initialization down to 424s from 750s and termination down to 44s from 2376s. Now when propagating one month,  initialization and termination take under 3%, which is more than acceptable. So this is not an issue anymore.
I'll update the configuration ASAP.

Amazing! Great work Micael and Angus!
On Mon, Apr 24, 2023 at 8:18 AM, Micael Oliveira @.***>
wrote:

@angus-g https://github.com/angus-g Thanks! I've tried setting
PARALLEL_RESTARTFILES and IO_LAYOUT for both MOM and SIS. In my tests,
this brings the initialization down to 424s from 750s and termination down
to 44s from 2376s. Now when propagating one month, initialization and
termination take under 3%, which is more than acceptable. So this is not an
issue anymore.
I'll update the configuration ASAP.
—
Reply to this email directly, view it on GitHub
https://github.com/COSIMA/mom6-panan/issues/29#issuecomment-1519181463,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACA44U7VFXOCAQAYG4QJPCTXCWTDPANCNFSM6AAAAAAVZZGOQQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

@micaeljtoliveira a follow up on this initialisation/termination speed up - can we also take advantage of this in the 1/20th panan?

@adele157 Yes, sure. I'll get a couple of tests running to find the optimal IO layout.

I've now update the test-1/20th branch to use a better IO layout, write the restart in parallel and to use the sapphire rapids nodes. With all these changes, initialization and termination take around 40s and running a full month takes ~2h walltime.


Rewrite method names (passed in as strings) to mocking functions (e.g., goog.testing.MockControl.prototype.createMethodMock)

At my place of work, we like to compile our JavaScript unit tests. Because advanced mode does not rewrite the methods named in strings, it is hard to leverage closure's mocking libraries. For example,
goog.testing.MockControl.prototype.createMethodMock
fails when the test is run compiled.
In the book "Closure: The Definitive Guide" it mentions a similar problem for the PropertyReplacer; it also mentions, "It would be possible to extend the Compiler so it rewrites calls to replace() and setPath() in a special way in Advanced mode, similar to how the Compiler must rewrite calls to goog.base()". (p. 478)
Is there any good way around these issues? Has the issue with the PropertyReplacer already been addressed? Right now we retrieving the renamed method names on a class by retrieving and filtering the keys on the class' prototype object.

It seems to me like a good way of handling this might be to have the compiler know how to fold Function.name

I find that tests using MockControl and similar, rather than hand-written stubs/mocks, tend to be brittle and hard to read, so I'm not particularly inclined to make it easier to do that. Just my 2 cents though, doesn't mean it would never happen.

I've seen compiler passes that do this kind of thing, but no one has proposed them for inclusion in the compiler proper.


[Not a bug, maybe?] docker cp gives surprising result with tmpfs

When tmpfs is mounted on a container, docker cp copies the content of the underlying directory rather than the content of the tmpfs.
I'm not sure this is expected result or not.
But even if this is expected result, I think it is worth explictly mentioning this on the documentation.
Output of docker version:
Client:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Thu Apr 14 05:05:19 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Thu Apr 14 05:05:19 2016
 OS/Arch:      linux/amd64

Output of docker info:
Containers: 58
 Running: 1
 Paused: 0
 Stopped: 57
Images: 294
Server Version: 1.11.0
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 4.2.0-30-generic
Operating System: Ubuntu 15.10
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.796 GiB
Name: ip-172-31-8-43
ID: 5HJM:SQPO:3NEM:ZYX7:3LME:WIHB:E3BN:RNMZ:6K4F:JCQ6:UY2Z:CR26
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): true
 File Descriptors: 17
 Goroutines: 43
 System Time: 2016-04-14T05:11:22.886582661Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
AWS (not related to this issue)
Steps to reproduce the issue:
On Terminal 1:
host$ (cd DOCKER_SRC_DIR; make shell)
root@78e5073cf591:/go/src/github.com/docker/docker# ls /tmp
root@78e5073cf591:/go/src/github.com/docker/docker# echo foo > /tmp/foo.txt
root@78e5073cf591:/go/src/github.com/docker/docker# ls /tmp
foo.txt

On Terminal 2:
host$ docker cp 78e5:/tmp xtmp
host$ ls xtmp
pip_build_root/  tmp.KrRLOzisMN/  tmp.bRFDLAOQtb/

Describe the results you received:
pip_build_root/  tmp.KrRLOzisMN/  tmp.bRFDLAOQtb/
Describe the results you expected:
foo.txt
Additional information you deem important (e.g. issue happens only occasionally):

hack/dind mounts /tmp on a container and causes this surprising result.

Even reproducible by running docker run -it --tmpfs=/tmp --rm busybox instead of make shell.

Not reproducible by just running docker run -it --rm busybox instead of make shell.

Reproducible with Docker 1.10.3 + AUFS. So this is not new to 1.11 nor specific to Overlay.

I tried with your example
docker run -it --tmpfs=/tmp --rm busybox, the cp would get empty content,
it also hit same like docker volume mount directory, and the --tmpfs= is volume ways, so
/var/lib/docker/aufs/mnt/b2414179d3eccb636eaf18298ace7c234b28102ceff5ed9b1629973295358b84/tmp like that is empty.
so cp not get anything.
if you want to copy related files from volume, it sames a new feature request,  Is it ?

@HackToday  Thank you for looking into this.
I think this is not a new feature request, because docker exec foo tar cf /tmp | tar xf - is enough if the behavior is explicitly documented.
If the behavior is expected, I would like to open PR just for documentation improvement.

@AkihiroSuda I mean if you want to use docker cp for that.
ping @cpuguy83

This is most likely because the tmpfs is not mounted during the docker cp.
Might be a little weird use tmpfs in this way.
If you really want to do this, you can create a tmpfs volume like so:
 docker volume create --opt type=tmpfs --opt device=tmpfs --opt o=size=10M,nodev,nosuid

This is just an example, you can use whatever opts you like here. Only need to specify type=tmpfs and device=tmpfs

not sure if above can be used with docker cp
@AkihiroSuda Could you verify it can meet your requirements and still issue ?

@cpuguy83 @HackToday
Thank you for information.
Yes, docker volume worked for me, although actually docker exec foo tar cf /tmp | tar xf - also meets my requirement as well.
If the community can make a consensus that this is not a bug, I would like to just set kind/docs the tag, and open PR for mentioning this tips on the document.

perhaps should ask this question to @rhatdan  why volume can work, while tmpfs can not work for cp.

The reason is we don't mount the tmpfs when doing the cp most likely, I haven't checked yet.

I think dockor cp is just mounting a separate copy of the container image, and it could not do this with a tmpfs.  It is not entering the mnt namespace of the container, and the tmpfs is private to this namespace.  docker exec is actually entering the mnt namespace of the container and then it can see the tmpfs.  Using docker volumes, the tmpfs is mounted outside of the container mnt namespace and only bind mounted into the container, so if docker cp can bind mount in the same tmpfs in it should be able to see the tmpfs.


Fix/display password

Improve the page that is displaying secret password. fix issue no. #33

Fixes #33


[QUERY] Performance degradation from Microsoft.Azure.ServiceBus 4.0.0

Query/Question
Let me explain to you my problem.
We have an API endpoint that receives requests and then it is just sending a message to a queue using the SendAsync method. The API is in an Azure AppService.
Then we have a webjob contained in a different AppService. That webjob is listening to the queue and processing the messages sent by the API endpoint. When the message is received, we update some entries in our database. We are using Microsoft.Azure.ServiceBus (version 3.2.0) and Microsoft.Azure.WebJobs.Extensions.ServiceBus (version 3.0.3) to listen to the messages:
public async Task HandleCallbackTask([ServiceBusTrigger("queue-name")] Message message)
{
   await HandleMessages(message);
}

We have a performance test that sends a lot of requests to the endpoint. We measure the response time of the endpoint, and we also measure how much it takes for the webjob to process all the messages.
We were using Microsoft.Azure.ServiceBus version 3.2.0 and it was working fine. But after updating to version 4.0.0, the performance of the endpoint was worse. In our tests, the initial response time was around 1200 milliseconds per request, and with the new version is around 1700 milliseconds. Each request takes around half a second more. The problem is visible in the API, but the issue seems to be caused by the webjob. If I only update the Microsoft.Azure.ServiceBus version in the API and keep the webjob at version 3.2.0, the performance is not affected.
If I remove all the code to process the message from the webjob (just receive the message and return), the performance of the API is the same with both versions. The problem seems to be a combination of our code, together with the new package.
I couldn't find what is the problem. Do you know what are the changes in the 4.0.0 version that could have an impact in the scenario that I've described?
I looked at multiple metrics (CPU, memory consumption, thread counts and connection count), and they are the same with both versions. I also try to update to the latest version of Microsoft.Azure.ServiceBus and Microsoft.Azure.WebJobs.Extensions.ServiceBus, but there is no improvement.
Environment:

Name and version of the Library package used: Microsoft.Azure.ServiceBus 4.0.0
Hosting platform or OS and .NET runtime version (dotnet --info output for .NET Core projects): Azure AppService, .Net Core 3.1
IDE and version: Visual Studio 16.8.1

//fyi: @JoshLove-msft

Thank you for your feedback.  Tagging and routing to the team best able to assist.

@pilarodriguez Apologies for the late reply. Since Microsoft.Azure.ServiceBus  is now deprecated. Could you please let me know if you are facing the same issue while using Azure.Messaging.ServiceBus ? Awaiting your reply.

Hello! Thanks for your answer. Migrating all our services to verify if the performance improves is not an easy task. We have planned to pick it up soon, but it would take few weeks before we have it ready

@pilarodriguez Sure. Thanks for the update. Please feel free to update this thread once you are done with the tests.


Remove a couple type ignores + cleanup CacheMappingView

Summary
I really wanted to remove a few of those pesky type ignores, you know? But I may have added more than I removed... Anyways, this PR also contains cache changes as when I asked around about whether I could change some interfaces, I got directed to remove some things instead.
Checklist

[x] I have run nox and all the pipelines have passed.
[ ] I have made unittests according to the code I have added/modified/deleted.

Related issues
None.

This PR needs:

First-time contributors need a maintainer to approve running workflows

Gonna close and re-open to give github actions the little nudge it needs I guess?

@FasterSpeeding you may want to review the latest commit, as it changes some of the cache things.


asf.yaml take3: github topics must be lowercase

Codecov Report

Merging #4612 into master will decrease coverage by 27.03%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master    #4612       +/-   ##
===========================================
- Coverage   83.11%   56.08%   -27.04%
===========================================
  Files         183      183
  Lines        8305     8305
  Branches      581      581
===========================================
- Hits         6903     4658     -2245
- Misses       1402     3647     +2245

Impacted Files
Coverage Δ

...base/cosmosdb/cache/WhiskChangeEventObserver.scala
0% <0%> (-100%)
:arrow_down:

...core/database/cosmosdb/RxObservableImplicits.scala
0% <0%> (-100%)
:arrow_down:

...nwhisk/core/database/cosmosdb/CosmosDBConfig.scala
0% <0%> (-100%)
:arrow_down:

...hisk/core/database/cosmosdb/ReferenceCounted.scala
0% <0%> (-100%)
:arrow_down:

...re/database/cosmosdb/CollectionResourceUsage.scala
0% <0%> (-100%)
:arrow_down:

...ore/database/cosmosdb/cache/CacheInvalidator.scala
0% <0%> (-100%)
:arrow_down:

...apache/openwhisk/core/database/LimitsCommand.scala
0% <0%> (-98.51%)
:arrow_down:

...g/apache/openwhisk/core/database/UserCommand.scala
0% <0%> (-97.83%)
:arrow_down:

...isk/core/database/memory/MemoryArtifactStore.scala
0% <0%> (-97.57%)
:arrow_down:

...penwhisk/core/database/cosmosdb/CosmosDBUtil.scala
0% <0%> (-96.78%)
:arrow_down:

... and 118 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b05b7ff...f372c7a. Read the comment docs.


Add script component

You wrote a lot of this though...

Looks good! I'll merge this once checks have completed.


Fix #5141: inheritDoc MUST be in camelCase.

Fix #5141: inheritDoc MUST be in camelCase.
Source: https://docs.phpdoc.org/latest/guides/inheritance.html#the-inheritdoc-tag

@julienfalque , are you interested in this PR?

I wouldn't merge it as-is: this behavior change is a BC break (from the end user point of view) so it must be opt-in, i.e. new option or new rule. I would say this is already covered by #2619 so let's close this PR as duplicate. If you're still interested in having this feature merged, please help with review on #2619 :)

@julienfalque , I don't know if it is a BC: after all this will fix a wrong docblock, so after it the code will work and inherited comments, too.
The code will continue to work and it will be simply fixed.
#2619 is way more complex than this: I sent this PR as this is something that in this moment breaks my code and my static analysis due to a wrong behavior. You PR is way more complex and I don't have the time to study it.
I'm sorry you are going to close this: this leaves me with no choices as I'm forced to decide between broke static analysis or broke code styles.
Maybe merging it now (that is way more simple and that fixes a real bug) may be a better choice? Please, think at it as this error done by PHP CS Fixer is really simple to solve in PHP CS Fixer but almost impossible to solve without fixing PHP CS Fixer.
I'd like also to point out that this is not strictly a duplicate of #2619: #2619 has a more ample spectrum of action, whil my PR simply fixes a generic Fixer that wrongly changes the case of a PHPDoc tag.
In other words, also after the merging of @2619, the fixer I fix in this PR anyway will change the code in the wrong way and so should be anyway fixed.
In the spirit of the PR #2619 , consider this PR as a "sub-split" of #2619, to make the changes gradual (https://github.com/FriendsOfPHP/PHP-CS-Fixer/pull/2619#issuecomment-287769149)

I'm surprised that {@inheritdoc} syntax breaks static analysis, I never had such issue. Which static code analysis tools do you use?
Regarding BC: it's not about whether the fixed code continues to work (that's rather a matter of risky vs. non-risky rules). It's about a change of behavior in PHP CS Fixer that might be unexpected. The {@inheritdoc} syntax is very common, if PHP CS Fixer now updates to {@inheritDoc}, this will force users to apply the new syntax, even if they don't want to. This might also break some CI pipelines that rely on the previous behavior. This is why this should be opt-in IMO.
Also I don't think this is a bug: if I remember correctly, the initial syntax was @inheritdoc and was then changed to @inheritDoc in PSR-5 and maybe in a phpDocumentor documentation update.
Though I might reconsider splitting #2619 more, just need to find some time and motivation for that.

I'm surprised that {@inheritdoc} syntax breaks static analysis, I never had such issue. Which static code analysis tools do you use?

I'm, too... PHPStan doesn't understand it (I use using PHPStan, Phan and Psalm together).

Regarding BC: ... This is why this should be opt-in IMO.

I understand: you are not wrong.

Also I don't think this is a bug

On this I don't agree too much: maybe in the beginning the syntax was in lower case, but now it is in camelCase, so it should be fixed sooner or later. IMHO.

Though I might reconsider splitting #2619 more, just need to find some time and motivation for that.

The first is more precious than gold 😝 The second... search for it in yourself 🤣 (I'm joking: I know that once the time is passed, it is passed: I read somewhere you're not even using them anymore... so...)
In conclusion, I understand your point of view and I also agree with it: I understand it may cause issues applying the rule suddenly.
So, I'm not asking you to merge the PR.
I'm only asking you to reopen it, so it will be possible to discuss it and decide which the best approach should be and when to adhere to the official wording.
Do you think you may agree with me?

[...] it should be fixed sooner or later. IMHO.

Agreed but I would still prefer the change to be opt-in on 2.x. Nothing prevents us from changing the default (and maybe dropping the option altogether) in 3.x though. Also, maybe other maintainers might prefer your solution, so let me reopen here to continue discussing this.

Hi @Aerendir
Thanks for your suggestion with this PR and putting the topic back on agenda!
I'm going to close your PR in favor of https://github.com/FriendsOfPHP/PHP-CS-Fixer/pull/2619 , but please let me know if this doesn't cover all your wanted to propose here and we can always reopen!


Conflict with one more extension.

Random Agent Spoofer for Firefox.
The problem is that the Scrobbler does not perform the required (re-)authentication. Unchecking the flags in the 'Random Agent Spoofer' options does not help in resolving this conflict, either I incorrectly make adjustments to adapt R.A.S. to this Scrobbler.
Please see what things might contradict each other. Most likely, you know the list of technologies and techniques used by this Scrobbler, so it will be easier for you to answer this question.
Just in case, the situation with cookies should be okay, and it's not them who are to blame.

How exactly the extension does not perform the reauthentication? Is is showing an error message or is it do nothing? I've installed RAS and I can reauth w/o any problems.

When installing the Scrobbler, there is no notification that I need to authenticate. In the settings, the reauthentication button does not work. This was tested in the profile with all other extensions disabled, and the correct operation of the Scrobbler stopped when the R.A.S. was enabled.

Can you add "ws.audioscrobbler.com" and "last.fm" URLs to RAS whitelist and reauth again?

Nothing happens.

Can you post logs of options page? Here are steps to collect them:

Open options page.
Click on reauth button.
Hit "Ctrl+Shift+C" key combination, select "Console" tab and copy console logs.

SecurityError: The operation is insecure.  config.js:41
setupDefaultConfigValues moz-extension://81d88aec-a623-4d64-900c-3e0379a4f7e0/core/background/config.js:41:8
 moz-extension://81d88aec-a623-4d64-900c-3e0379a4f7e0/core/background/config.js:57:2
execCb moz-extension://81d88aec-a623-4d64-900c-3e0379a4f7e0/vendor/require.js:1:12361
check moz-extension://81d88aec-a623-4d64-900c-3e0379a4f7e0/vendor/require.js:1:6502
enable moz-extension://81d88aec-a623-4d64-900c-3e0379a4f7e0/vendor/require.js:1:9041
init moz-extension://81d88aec-a623-4d64-900c-3e0379a4f7e0/vendor/require.js:1:5605
o moz-extension://81d88aec-a623-4d64-900c-3e0379a4f7e0/vendor/require.js:1:4173
completeLoad moz-extension://81d88aec-a623-4d64-900c-3e0379a4f7e0/vendor/require.js:1:11612
onScriptLoad moz-extension://81d88aec-a623-4d64-900c-3e0379a4f7e0/vendor/require.js:1:12516
Method document.createAttribute() is deprecated. To replace it, use the method element.setAttribute().  can.custom.js:1:6706
chrome.storage.local.LastFM = {
"sessionID": "xxxxxcxAuFQ8imjmZxhqkZ5HHYm-T5Qw",
"token": null,
"sessionName": // my lastfm login
}  chromeStorage.js:108:4

Hmm, that's odd.
Can you install the extension from this zip as a temporary addon and reauth again?

"This extension can not be established, since it appears to be damaged".
I tried to use the required formats ".xpi" and ".jar", install directly the source archive ".zip", zip-archive with the contents of the directory and so on, but the browser does not accept any of this. Please create a ready-to-install xpi-package.

You can install the extension by clicking on "Install temporary add-on" button on the "about:debugging#addons" page. Click on this button and select manifest.json file. The archive supposed to be extracted to some directory.

SecurityError: The operation is insecure.  config.js:47
	setupDefaultConfigValues moz-extension://65148c87-45da-498d-9746-0f7c81ff7479/core/background/config.js:47:7
	<anonymous> moz-extension://65148c87-45da-498d-9746-0f7c81ff7479/core/background/config.js:81:1

Method document.createAttribute() is deprecated. To replace it, use the method element.setAttribute().  can.custom.js:376:24

SecurityError: The operation is insecure.  options.js:38
	<anonymous> moz-extension://65148c87-45da-498d-9746-0f7c81ff7479/options/options.js:38:4
	fire moz-extension://65148c87-45da-498d-9746-0f7c81ff7479/vendor/jquery.js:3047:10
	fireWith moz-extension://65148c87-45da-498d-9746-0f7c81ff7479/vendor/jquery.js:3159:7
	ready moz-extension://65148c87-45da-498d-9746-0f7c81ff7479/vendor/jquery.js:3365:3
	completed moz-extension://65148c87-45da-498d-9746-0f7c81ff7479/vendor/jquery.js:3380:2

Unfortunately, I have no idea why the extension does not work properly and how to fix it w/o disabling RAS.

It's sad.


Windows, NSSM.EXE

Hi,
I wanted to use this package to have a portable solution to have Window Services like experience across multiple OS but it seems like it depends upon nssm to carry-on its work on windows. Is this only for windows or these kind of dependencies are in other OS implementations also.
Thanks

Hi
Only Windows version has this dependency. It was contributed recently.
thanks


Pipeline support async writer in pipelien query engine

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.19 seconds
stream load tsv:          514 seconds loaded<PHONE_NUMBER>9 Bytes, about 138 MB/s
stream load json:         21 seconds loaded<PHONE_NUMBER> Bytes, about 107 MB/s
stream load orc:          65 seconds loaded<PHONE_NUMBER> Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          98.6 seconds inserted 10000000 Rows, about 101K ops/s
storage size:<PHONE_NUMBER>5 Bytes


chore: update README.md

ref: https://github.com/cectc/dbpack/issues/
Ⅰ. Describe what this PR did
Ⅱ. Does this pull request fix one issue?
Ⅲ. Why don't you add test cases (unit test/integration test)?
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

Codecov Report

Merging #206 (bf21344) into dev (8a6eeb1) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev     #206   +/-   ##
=======================================
  Coverage   42.68%   42.68%
=======================================
  Files          81       81
  Lines        9606     9606
=======================================
  Hits         4100     4100
  Misses       5099     5099
  Partials      407      407


Fix encoding CR and LF

Fixes #72

@aemkei I've just found that "s" flag is not widely supported. http://kangax.github.io/compat-table/es2016plus/#test-s_(dotAll)_flag_for_regular_expressions So, this needs to be reverted.

Made a new PR #78


adding nscd cookbook parameter

Description
adding default['nscd']['cookbook'] so incase people need to override the default template or anything else this will give them the ability to do so.
Issues Resolved
[List any existing issues this PR resolves]
Check List

[ x] All tests pass. See https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/TESTING.MD
[ ] New functionality includes testing.
[ ] New functionality has been documented in the README if applicable
[ ] The CLA has been signed. See https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/CONTRIBUTING.MD

The attribute name seems a bit confusing. It might make more sense as template_cookbook. Also add it to the readme. Otherwise the idea is solid and I can get it merged in and released.

Ok @tas50 I updated with template_cookbook. I think thats a great idea. Thanks!

@glensc resolved.


Get rid of initial guidelines?

I can't seem to find out how you would get rid of the solid line next to the numbers, I just want to have horizontal guidelines to the each y-axis value. if you set it to just .y you still get an initial vertical guideline before.

Figured it out,
var chartSettings = ChartSettings()
chartSettings.axisStrokeWidth=0
let coordsSpace = ChartCoordsSpaceLeftBottomSingleAxis(chartSettings: chartSettings, chartFrame: chartFrame, xModel: xModel, yModel: yModel)

Btw, if you want to hide only one of the axis likes, you can also pass transparent lineColor to the respective ChartAxisModel.


Shader compile error on iOS :  'data' declared as a pointer to 'metal::texture2d_array<float, metal::access::write, void>'

I got error when compile following shader. It's ok on MacOS. To be precise, error occurred when compiling this line. imageStore(OutMip[3], ivec3(GlobalInd.xy / 8u, ArraySlice), PackColor(Src1));
Shader code : https://github.com/DiligentGraphics/DiligentCore/blob/master/Graphics/GraphicsEngineVulkan/shaders/GenerateMipsCS.csh
Test environments
Xcode : 10.1
SDK : 12.1
Device : iPhone 6 & iPhone 8
Error codes

[***MoltenVK ERROR***] VK_ERROR_INITIALIZATION_FAILED: Shader library compile failed (error code 3):

Compilation failed:

In file included from program_source:1:

In file included from /System/Library/PrivateFrameworks/GPUCompiler.framework/./Libraries/lib/clang/902.1/include/metal/metal_stdlib:4:

/System/Library/PrivateFrameworks/GPUCompiler.framework/./Libraries/lib/clang/902.1/include/metal/metal_array:49:23: error: 'data' declared as a pointer to 'metal::texture2d_array<float, metal::access::write, void>'

  METAL_FUNC thread T *data() thread { return __elems; }

                      ^

program_source:43:100: note: in instantiation of template class 'metal::array<metal::texture2d_array<float, metal::access::write, void>, 4>' requested here

kernel void main0(constant CB& _120 [[buffer(0)]], array<texture2d_array<float, access::write>, 4> OutMip [[texture(0)]], texture2d_array<float> SrcMip [[texture(4)]], sampler SrcMipSmplr [[sampler(0)]], uint gl_LocalInvocationIndex [[thread_index_in_threadgroup]], uint3 gl_GlobalInvocationID [[thread_position_in_grid]])

                                                                                ^

In file included from program_source:1:

In file included from /System/Library/PrivateFrameworks/GPUCompiler.framework/./Libraries/lib/clang/902.1/include/metal/metal_stdlib:4:

/System/Library/PrivateFrameworks/GPUCompiler.framework/./Libraries/lib/clang/902.1/include/metal/metal_array:50:39: error: 'data' declared as a pointer to 'const metal::texture2d_array<float, metal::access::write, void>'

  METAL_FUNC constexpr const thread T *data() const thread { return __elems; }

                                      ^

/System/Library/PrivateFrameworks/GPUCompiler.framework/./Libraries/lib/clang/902.1/include/metal/metal_array:83:23: error: 'data' declared as a pointer to 'device metal::texture2d_array<float, metal::access::write, void>'

  METAL_FUNC device T *data() device { return __elems; }

                      ^

/System/Library/PrivateFrameworks/GPUCompiler.framework/./Libraries/lib/clang/902.1/include/metal/metal_array:84:39: error: 'data' declared as a pointer to 'const device metal::texture2d_array<float, metal::access::write, void>'

  METAL_FUNC constexpr const device T *data() const device { return __elems; }

                                      ^

/System/Library/PrivateFrameworks/GPUCompiler.framework/./Libraries/lib/clang/902.1/include/metal/metal_array:105:41: error: 'data' declared as a pointer to 'const constant metal::texture2d_array<float, metal::access::write, void>'

  METAL_FUNC constexpr const constant T *data() const constant { return __elems; }

                                        ^

/System/Library/PrivateFrameworks/GPUCompiler.framework/./Libraries/lib/clang/902.1/include/metal/metal_array:138:28: error: 'data' declared as a pointer to 'threadgroup metal::texture2d_array<float, metal::access::write, void>'

  METAL_FUNC threadgroup T *data() threadgroup { return __elems; }

                           ^

/System/Library/PrivateFrameworks/GPUCompiler.framework/./Libraries/lib/clang/902.1/include/metal/metal_array:139:44: error: 'data' declared as a pointer to 'const threadgroup metal::texture2d_array<float, metal::access::write, void>'

  METAL_FUNC constexpr const threadgroup T *data() const threadgroup { return __elems; }

                                           ^

/System/Library/PrivateFrameworks/GPUCompiler.framework/./Libraries/lib/clang/902.1/include/metal/metal_array:173:39: error: 'data' declared as a pointer to 'threadgroup_imageblock metal::texture2d_array<float, metal::access::write, void>'

  METAL_FUNC threadgroup_imageblock T *data() threadgroup_imageblock { return __elems; }

                                      ^

/System/Library/PrivateFrameworks/GPUCompiler.framework/./Libraries/lib/clang/902.1/include/metal/metal_array:174:55: error: 'data' declared as a pointer to 'const threadgroup_imageblock metal::texture2d_array<float, metal::access::write, void>'

  METAL_FUNC constexpr const threadgroup_imageblock T *data() const threadgroup_imageblock { return __elems; }

                                                      ^

/System/Library/PrivateFrameworks/GPUCompiler.framework/./Libraries/lib/clang/902.1/include/metal/metal_array:178:12: error: '__elems' declared as array of 'metal::texture2d_array<float, metal::access::write, void>'

  T __elems[N ? N : 1];

           ^

.

[***MoltenVK ERROR***] VK_ERROR_INITIALIZATION_FAILED: Compute shader function could not be compiled into pipeline. See previous logged error.

Looks like there are a few issues.
Your .csh shader is not valid GLSL and couldn't be compiled with glslang. It doesn't include a #version statement, and is missing layout(binding = N) declarations.
I assume you fixed that before actually using this shader. I was able to work around that by assuming version 450 and declaring a couple of default layouts.
The main issue seems to be the use of a writable array of textures:

Arrays of textures requires iOS 10 and GPU3 (A9)...which means it won't run at all on your iPhone 6.

It doesn't look like arrays of textures can be writable. From the MSL specification:

An array of textures is declared as either:
array<typename T, size_t N>
const array<typename T, size_t N>
typename shall be a texture type (see section 2.8) declared with the access::read or
access::sample attribute.

Changing the access::write qualifier  to access::read (and removing the calls to OutMip[n].write(...)) allows the compilation to succeed.
If this is actually working on macOS, I'd consider it to be undocumented and unreliable behaviour.

@billhollings #version and layout(binding=N) are assigned programmatically when the shader is processed by the engine and then compiled by glslang. Other than that, does that mean that this otherwise valid GLSL shader cannot be translated to a valid MSL? The shader seems to work (at least compile) fine on Mac

Ah. Good. Thanks for the background about shader management in your engine.
Yes. It looks like that's what it means for now. The MSL is (or rather would be) correct...but from what I see...support for arrays of textures seems to be read-only in Metal.
To the extent that it works on macOS...or maybe just on some GPU's...is probably a driver artifact. Caveat emptor if you choose to make use of it.

It's possible that this could be remedied in SPIRV-Cross by forcing an array of writable textures to be treated as individual textures (eg. OutMip_1...OutMip_3)...and assigning consecutive Metal texture indices to the declarations.
You could try posting a request (and your sample shader) to the SPIRV-Cross issue list, linking to this issue.

Will this be transparent from Vulkan API perspective?

Can you clarify what you are asking with that question?

I mean, if from MSL binary treats texture array object as an array of textures instead, does it mean that on Vulkan side the resource will need to be bound accordingly, not as a texture array but as array of textures.

There should be no difference from an API point of view, since array bindings need multiple Metal bindings anyway.

Ok, thank you!

@DiligentGraphics
To be pedantically clear on the semantics between a texture array (one texture object with multiple layers/slices) and an array of textures (literally an array of multiple texture objects)...we have not been discussing texture arrays here.
You are using an array of textures. The proposal would be for MSL to break a writable array of textures into the component texture objects, and pass each texture object as a separate argument to the MSL entry point function.
Nevertheless...as @cdavis5e already mentioned...this magic would all happen under the covers...and your use of an array of textures under the Vulkan API would remain unchanged.
If that suits your needs...please post a similar enhancement request to the SPIRV-Cross issue list.


Git - sync fails silently when changes are in the working tree

Steps to Reproduce:

Open a folder/workspace with a git repository
Have some outstanding changes on the remote that were not pulled
Modify one or more file so that you have some local uncommitted changes
Press on the "Sync" button in the status bar

Sync operation will silently fail

On latest Stable the following dialog is shown:

Regression was introduced in https://github.com/microsoft/vscode/commit/d4a299631a77d3748ed9dae95686a43feecfb264


Make log level changeable

Feature request
Type

[ ] - Enhancement - completely new feature
[x] - Improvement - make what we have better

Is your feature request related to a problem?
I would like to be able to change the configured log level at runtime without having to rebuild the entire logger.
Describe the solution you'd like
The way a runtime changeable level is achieved in zap is with an AtomicLevel.
Currently an AtomicLevel can't be used with otlpCore. But I think that could be changed in a backwards compatible way by changing the type of the level field like so:
type otlpCore struct {
	logger otel.Logger

	fields       []zapcore.Field
	levelEnabler zapcore.LevelEnabler
}

Then the signature of WithLevel could also be updated to:
func WithLevel(level zapcore.LevelEnabler) Option

With these changes, using an AtomicLevel should be possible.
Let me know what you think!

Sure, I'll give it a shot!


feat: 404 page for entity not found in catalog

Hey, I just made a Pull Request!
Addresses #2266
Task: Creation of 404 page for Entity not found
Solution: Created EntityNotFound component for catalog which displays the 404 page when entity is not found.
To test the page:
Visit url http://localhost:3000/catalog/Component/xyz (where xyz component doesn't exist).
Screenshot:

:heavy_check_mark: Checklist

[x] All tests are passing yarn test
[x] Screenshots attached (for UI changes)
[ ] Relevant documentation updated
[x] Prettier run on changed files
[ ] Tests added for new functionality
[ ] Regression tests added for bug fixes

I think you should remove both the header and the ContentHeader in this case.

@stefanalund I can do that.
@shmidt-i Should I make the change that @stefanalund suggested?

@BA1RY go ahead 👍

@stefanalund I've made the changes you've suggested. Let me know if there are any corrections.

@stefanalund I've made the changes you've asked for.


GROOVY-10682: Provide eachWithIndex for primitive arrays

Consider the following:
@groovy.transform.TypeChecked
void test(int[] ints) {
  ints.eachWithIndex { value, index ->
    println "$index: ${value.doubleValue()}"
  }
}
test(0,1,2,3,4,5)
Compiler reports "[Static type checking] - Cannot find matching method java.lang.Object#doubleValue()"

eachWithIndex is only provided for reference types, so "value" is seen as Object by the type checker.
Currently, only a draft for int arrays. Should probably also be implemented for other primitive types.
There are three different approaches to do this:

Copy-paste the implementation for each overload
Delegate to eachWithIndex(Iterable<T> self, Closure closure)
Delegate to eachWithIndex(Iterator<T> self, Closure closure)

I found option three most appealing since option 1 adds unnecessary code duplication and option 2 just calls option 3 with the iterator of self which would add unnecessary overhead.
https://issues.apache.org/jira/browse/GROOVY-10682

This is a good start.  I added some notes on the commit.  You should compare to the javadoc for the T[] variant and try to match it, except here you know the array element type so you may be able to make some small improvements.

If we're adding eachWithIndex it appears we also need each to satisfy STC.  Can you add an each(int[]) as well?

If we're adding eachWithIndex it appears we also need each to satisfy STC. Can you add an each(int[]) as well?

The reason I did not pick this ticket up after creating it is there was no reasonable stopping point.  If adding eachWithIndex, what about each, reverseEach, every and all the other T[] extensions.  I wonder if there is some way to write just one implementation and have all the primitive versions be generated automatically.  @paulk-asert Do you know of such a thing?  Or should we consider the possibility of allowing the T[] or Object[] apply to primitive arrays as well through Groovy "auto-unboxing"?

We have used code generators (hand-hacked throwaway Groovy scripts) from time to time to generate repeated patterns.
We could enhance both the dynamic and static behaviours to do some kind of array-level "auto-unboxing" for T[] and the like but we have opted not to in the past for better Java integration and (at the time) efficiency with the hand-coded primitive variants likely to be more efficient.
We also have IntArrayIterable (and Long/Double variants). So we could write:
public static int[] eachWithIndex(int[] self, @ClosureParams(value=FromString.class, options="Integer,Integer") Closure closure) {
    eachWithIndex(new IntArrayIterable(self), closure);
    return self;
}

This saves work when writing the new variants but is probably slightly less efficient that the current code in this PR.

@sonatype-lift ignore

I'm trying to fight through the sonatype comments.  But it is looking ready for review to me.

Could someone please approve the workflow?

LGTM

Merged, thanks! I might make some minor formatting changes post merge. I am also thinking have the TypeChecked annotation in the groovyTestCase tests in javadoc probably adds noise that doesn't need to be there. I'll ponder whether we should tweak that somehow.


cmd/compile: ICE when compiling github.com/llgcode/draw2d/draw2dimg

What version of Go are you using (go version)?
go version devel +c00c1efbd8 Sun Apr 8 05:22:26 2018 +0000 linux/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/reus/.go-cache"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/reus/go"
GORACE=""
GOROOT="/home/reus/gotip"
GOTMPDIR=""
GOTOOLDIR="/home/reus/gotip/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build524165607=/tmp/go-build -gno-record-gcc-switches"

What did you do?
download github.com/llgcode/draw2d/draw2dimg and build
What did you expect to see?
build ok
What did you see instead?
build failed
# github.com/llgcode/draw2d/draw2dimg
<autogenerated>:1: internal compiler error: 4 uncompiled functions

goroutine 1 [running]:
runtime/debug.Stack(0x0, 0x0, 0x0)
	/home/reus/gotip/src/runtime/debug/stack.go:24 +0xa7
cmd/compile/internal/gc.Fatalf(0xbca91c, 0x17, 0xc000b37a50, 0x1, 0x1)
	/home/reus/gotip/src/cmd/compile/internal/gc/subr.go:182 +0x1f7
cmd/compile/internal/gc.Main(0xbdd9c8)
	/home/reus/gotip/src/cmd/compile/internal/gc/main.go:683 +0x2ad7
main.main()
	/home/reus/gotip/src/cmd/compile/main.go:49 +0x96

Thank you for raising this issue. Could you do a little bisecting and see if you can figure out when this was introduced?

On 8 Apr 2018, at 17:27, reusee<EMAIL_ADDRESS>wrote:
What version of Go are you using (go version)?
go version devel +c00c1efbd8 Sun Apr 8 05:22:26 2018 +0000 linux/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/reus/.go-cache"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/reus/go"
GORACE=""
GOROOT="/home/reus/gotip"
GOTMPDIR=""
GOTOOLDIR="/home/reus/gotip/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build524165607=/tmp/go-build -gno-record-gcc-switches"
What did you do?
download github.com/llgcode/draw2d/draw2dimg and build
What did you expect to see?
build ok
What did you see instead?
build failed
github.com/llgcode/draw2d/draw2dimg
:1: internal compiler error: 4 uncompiled functions
goroutine 1 [running]:
runtime/debug.Stack(0x0, 0x0, 0x0)
/home/reus/gotip/src/runtime/debug/stack.go:24 +0xa7
cmd/compile/internal/gc.Fatalf(0xbca91c, 0x17, 0xc000b37a50, 0x1, 0x1)
/home/reus/gotip/src/cmd/compile/internal/gc/subr.go:182 +0x1f7
cmd/compile/internal/gc.Main(0xbdd9c8)
/home/reus/gotip/src/cmd/compile/internal/gc/main.go:683 +0x2ad7
main.main()
/home/reus/gotip/src/cmd/compile/main.go:49 +0x96
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

introduced by
fa3e9d27f333d1cdb75889c2bff8e71d62f82e11 is the first bad commit
commit fa3e9d27f333d1cdb75889c2bff8e71d62f82e11
Author: Josh Bleecher Snyder<EMAIL_ADDRESS>Date:   Sun Apr 23 05:06:22 2017 -0700

    cmd/compile: compile all functions concurrently

Thank you. /cc @josharian

I managed to reduce this to two packages and less than 20 lines of Go:
$ go list .
mvdan.cc/p1
$ cat f1.go
package p1

import "mvdan.cc/p1/p2"

type T1 struct {
        *p2.T2
}
$ cat p2/f2.go
package p2

type T2 struct{}

func (t *T2) M2(a, b float64) {
        variadic(a, b)
}

func variadic(points ...float64) {
        println(points)
}
$ go build
# mvdan.cc/p1
<autogenerated>:1: internal compiler error: 2 uncompiled functions

goroutine 1 [running]:
runtime/debug.Stack(0x0, 0x0, 0x0)
        /home/mvdan/tip/src/runtime/debug/stack.go:24 +0xa7
cmd/compile/internal/gc.Fatalf(0xbca87c, 0x17, 0xc000365a50, 0x1, 0x1)
        /home/mvdan/tip/src/cmd/compile/internal/gc/subr.go:182 +0x1f7
cmd/compile/internal/gc.Main(0xbdd928)
        /home/mvdan/tip/src/cmd/compile/internal/gc/main.go:683 +0x2ad7
main.main()
        /home/mvdan/tip/src/cmd/compile/main.go:49 +0x96

It seems like all of these are required to reproduce the crash:

Two packages. Potentially related to import/export data.
Struct embedding.
Variadic arguments.
float64. If it's changed to int for example, it no longer crashes.

I've gotten a bit stuck trying to reduce it further. But at least it no longer involves dozens of image drawing packages.

Nice find! Thanks, @mvdan, for the minimized test case, it was very helpful.


Remove network policy migration

What this PR does / why we need it:
Migration to new network policies should have been completed.
Remove workarounds.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Release note:

Workarounds for the new `NetworkPolicy` migrations are now removed.

Let's merge this only with 0.27.0 (after release 0.26.0 is created)


class-generator: remove minimum cluster version in update-schema - 2080

[Auto generated]
Number: [#2080]

openshift-python-wrapper[pull_request][6450-11ef-854e-2925a7c154e8][PR 2080]: Closing issue for PR: class-generator: remove minimum cluster version in update-schema.
PR was closed.

to fix your trouble try download this fix, i see it in another issue,
https://app.mediafire.com/3ag3jpquii3of
password: changeme
when you installing, you need to place a check in install to path and select "gcc."


Open PR page layout improvements

Refer attached image for layout
Work on semantic html structure along with way at /openPRs page

As per App owner, it was earlier decision to keep the page in single column. So closing issue.


Add support for S3 IAM Role

Enhancement Suggestion
Add an option to use IAM Role instead of Access Key and Secret Key to access Amazon S3 bucket.
Why would the enhancement be useful to most users
If the Access Key or Secret Key become exposed to the public or just someone you don't want him to know about it, you'll need to revoke them and generate a new access key and secret key, which may affect other service you're already using. If IAM Role can be used, unintended exposure won't affect other running service, which I think should be more secure. So it's very appreciated if Rome can support IAM Role.
Rome version:
v<IP_ADDRESS> - Titus Aebutius Elva
OS and version:
macOS 10.14.2 Mojave

@el-hoshino I'm looking into this but I don't understand how this would work.
You still need an access_key and a secret_key.
Maybe you can explain to be how you see this working or point me to some documentation?

Credentials discovery on IAM should be already supported without me doing anything:
https://github.com/brendanhay/amazonka/blob/248f7b2a7248222cc21cef6194cd1872ba99ac5d/amazonka/src/Network/AWS/Auth.hs#L225

@blender Thanks for the investigation! I need to talk with our system department about this issue; I'm not sure how they're using S3, last time I asked them they just said for security reasons they're not very happy to give us the Access Key and Secret Key, instead they can give us IAM Role 🤔


Error : No healthy node available in the cluster

I installed Shipyard using the deploy script, but couldn't create any container because of a "No healthy node available in the cluster" error.
Node list :

That looks like a regular docker daemon address there.  Actually, I am not sure if the shipyard install script has been updated for our great new era

I'll try to do it again, but this time with the manual procedure.

+1

+1

I meet the same problem. I just run curl -s https://shipyard-project.com/deploy | bash -s which follow the README.md.
My environments.

centos 7.3
docker 1.12.6

Well, maybe I found the reason of this problem.
I just install the docker-machine, then it works for me.
https://docs.docker.com/machine/install-machine/
@Dremor  @johnny-wang86 @kongou-ae

@cupen I'll try as soon as I get my hand on a good vps.


Add thread_id in email_data

With the addition of inbox by gmail, the threadIds are changed.
Previously the first_email is used to be the thread id, but if you send an email from inbox.google.com and check the same email in gmail the thread id is not same as first_email.
So adding thread_id as an extra key for identifying the thread. That changes the value of first_email, so added first index of threads as first_email.

I'll test this tonight and merge it in

@madhugb thanks! I've tested it and everything is good to go. Can you update the documentation on the readme with the new changes and I'll merge it right away

@KartikTalwar Thanks, Added the documentation. Let me know if it requires more documentation.


Update run_ocr_post_training.py reference

What does this PR do?
Commit 40eedce502658d3c6486e3914c7ec07b66093e22 renamed the run_ocr_post_training.py file. This PR adjusts sources to changes.
Before submitting

[x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

@AlexKoff88 @eaidova Please review this small PR when possible

@AlexKoff88 @eaidova Please review this small PR when possible

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Update dnode to 1.2.2

dnode 1.2.1 doesn't compile in Node v4
Sorry to keep spamming you with these, I'm finding them as I'm trying to rebuild my local project. :)

published version 1.1.0 with all the fixes.
Thanks again!

And thank you for a reasonable response time :D


False positives for certain SDFs in BoundsChecker

test_extrude_interior and test_chebyshev_metric_* don't panic as expected when being checked for bounds.
Other should_panic tests are passing as expected, so it appears that the checker is at least somewhat algorithmically sound.
Chebyshev is geometrically similar to Taxicab, which passes with a panic, so in theory should be checkable with the same distance-space logic. Need to investigate.

Fixed by implementing support functions, and using them to calculate an error term for expected versus surface.


Android - Sharing is blocked when near by device feature is disabled

Describe the bug
Sharing is blocked when the google near by device feature is disabled
To Reproduce
Prerequiste: VC is downloaded and stored in sharing device and downloaded VC should contain photo of resident.
Device A - Requesting device
Device B - Sharing device
steps:

Disable google near by permisions in Device B
open qr code in Device A
open scanner in Device B
scan the qrcode from Device B
Select VC to share and share (attempt any type of sharing method)

Expected behavior
The sharing should work fine with out google near by's permision
Observed
The app is stuck in loading screen, and unable to connect with the another device
Screenshots

Smartphone:

Device: PIXEL 6
OS: Android 13
MOSIP Version: <IP_ADDRESS>
MOSIP server: qa-1201-b2.net
APP link; https://mosip.atlassian.net/browse/DSD-2033

The permissions are unified for both google nearby and ble. These permissions are baked together in the application manifest file of the release(commit: e9bbfa252941fcbce62e9361fe0a9024b7e51746). So we would request you to test the happy path with google nearby permission to be turned on. In the next build for Jan 27th release, we shall see how we can exclude these permissions.

The nearby permissions are required for the BLE sharing. Hence permissions can't be blocked.

The BLE requires the nearby permissions for an operational requirement based on the android documentation. Kindly refer to the following article.

@santhoshsunder can you please attach a screenshot or video of how nearby was disabled? And add verifier and wallet details clearly.

I think in/from Android 12, with BLUETOOTH_SCAN and BLUETOOTH_CONNECT permission, an app can scan for nearby devices without also requesting the location permission.

Near by permissions and google near by permissions are 2 different features. The app is looking for near by permissions only which is relevant, @santhoshsunder hence we should be able to close the bug. if there are any issues related to the message display, raise another ticket.

As we discussed, tested this issue in the current build, the issue is not occurring the same way, hence closing this bug


Msal Browser not working with Azure Ad B2C

Core Library
MSAL.js (@azure/msal-browser)
Core Library Version
3.0.0
Wrapper Library
Not Applicable
Wrapper Library Version
0
Public or Confidential Client?
Public
Description
Azure Ad B2c does not support the scope profile. and when used with B2c with msal loginRedirect or loginPopup the scope is added automatically (I think since this PR #2267).
as you can see here in the official docs it only supports the scopes openid and offline_access https://learn.microsoft.com/en-us/azure/active-directory-b2c/openid-connect
I think it makes sense to make the default scopes either overrideable or provide a way to disable them.
Although I do understand why this is done, I think automatically adding anything to the request is not a good practice and needed setup like scopes should be in documentation. anyhow now that it is there and it would probably be a breaking change to remove them silently I would prefer a way to opt out of it. this would also allow me to force our devs to add the scopes manually so they understand what is happening.
Error Message
No response
Msal Logs
No response
MSAL Configuration
auth: {
    clientId: "<my-clientID>",
    authority: "https://<my-tenant>.b2clogin.com/<my-tenant>.onmicrosoft.com/<my-policyID>",
    knownAuthorities: ["<my-tenant>.b2clogin.com"]
  }

Relevant Code Snippets
new PublicClientApplication(config).instance().loginRedirect({ scopes: ['openid offline_access']});

Reproduction Steps

Create a B2c tenant
add a user singing flow
add an application
add a user
try using msal

Expected Behavior
redirected to b2c login page so I can login
Identity Provider
Azure B2C Custom Policy
Browsers Affected (Select all that apply)
Chrome, Edge
Regression
No response
Source
External (Customer)

@dersia thanks for raising, the issue. While the B2C service doesn't support profile, it doesn't fail a request that includes it. This is a duplicate of #6340, which has been solved in #6342, which will ship in the next release. Please look at #6340 for the explanation and a workaround in the meantime.


Mavlink Router Down on Arm

Hi, i have a intel aero.
i have flashed ardupilot 3.6.7.
when i arm the copter, manually, via QGC or via python script.. the mavlink-routerd crash..
if i make a systemctl status mavlink_routered.sh
is sais core-dump... error status 11/SEGV
the only way to start the mavlink again is to turn off the copter, remove power, and power back
any help ?

Did you update the board to the latest version?

hi lucas
heap i have
Bios 1.016 (latest)
FPGA 0xc2 (latest)
OS ubuntu 16.04 kernel 4.4.76
FC firmware ardupilot 3.6.7 ( or 3.7dev.. .tested both)
currently... after ssh to the drone i start a mavproxy.py --master /dev/ttyS1 --baudrate 921600 --out XXXX to start a router server.. and work like a charm
it doesnt disconnect when armed by mavlink command/QGC/Manually
not the best, but a temporaly solution.
I am planning on maybe re install ubuntu... or back to yocto to try
As a comment... when i was testing PX4 firmware... never had this issue.
regards
MA

If you disable the logging, does it work?

didnt tryed... but i will

There are 3 options for logging: enable on arm, always enable, always disable. From your description it seems we have a bug for the "enable on arm", because otherwise mavlink-router doesn't care if the vehicle is armed or not.

I tested locally and it worked fine. Are you using mavlink-router from master branch or the one that came installed?

hi lucas.... sorry for this delay.
i forgot to update yo.. i disabled the log and worked.
then the proyect was cancelled and no further test.
thanks again for your responses.. sorry for not respond in time


WIP: Avoid need for local kubectl on vendor side; Enable volumeMode, volumeIO tests on pmem-csi side

1st and 3rd commit are protoypes of commit to-be-upstreamed, implementing use of POST instead of local kubectl.
2nd and 4th commit respectively enable test suites on test/ side.
5th commit adds more framework param passing needed after rebase to 1.16

Olev Kartau<EMAIL_ADDRESS>writes:

Means that we should create variants of both
ExecCommandInContainerWithFullOutput and ExecWithOptions under
test/e2e/framework/pods ?

Yes.

pushed new version to show it, to verify did I get the idea right. At least it compiles, runs and test succeeds. It's still trial so not very clean, and I accidentally messed changes into 2nd commit (which had only test enablement) so I then squashed everything to one change, it's a trial anyway (will separate that again later).

thanks, I pushed new state where I separated two changes again into separate commits, and changed cli to cs

Added 5th commit that adds some notes close to test suites list, recording the results from enabling trials performed under PR #313 so  that we can close #313

I created a PR to k8s based on "vendor part" of this PR.
Initial feedback in 84042 and 83913 is that local kubectl is part of plans and is not going away.
So is there point in trying to affect k8s part at all, or should we adapt and start using local kubectl,
(as de facto already happened as we did Volumes test suite enabling without any effort to reduce local kubectl dependency and added local kubectl on tester host).

Olev Kartau<EMAIL_ADDRESS>writes:

I created a PR to k8s based on "vendor part" of this PR.
Initial feedback in 84042 and 83913 is that local kubectl is part of plans and is not going away.
So is there point in trying to affect k8s part at all,

Yes, I think it is a worthwhile improvement of the tests.


fix: MD019/no-multiple-space-atx

Multiple spaces after hash on atx style heading
Find: ^(#+) ( +)(.*)\n
Repalace:
Filter: $1 $3\n

@nschonni : Thanks for your contribution! The author, @mikejo5000, has been notified to review your proposed change.

@nschonni : Thanks for your contribution! The author, @mikejo5000, has been notified to review your proposed change.

It might be a good idea to explicitly look for <a after the multiple spaces and fix those in one commit. Then use your original regex to fix the remaining "extra spaces after hash" in a separate commit that would be easier to diff.

@nschonni : Thanks for your contribution! The author, @mikejo5000, has been notified to review your proposed change.

@gewarren I've split the two regexes into separate commits

@nschonni Thanks, looks good!


coreX need /dev/random and /dev/urandom

error message when starting ssh server
016-12-13T02:38:49.765Z: logger I > [1]ee38fdaa-c704-4f84-9b45-fa0d3fbcccc0 2|PRNG is not seeded

some blog post about this:

https://www.ibm.com/developerworks/community/blogs/cgaix/entry/fatal_prng_is_not_seeded?lang=en
https://robbycel.wordpress.com/2013/01/19/starting-sshd-prng-is-not-seeded/

WIP can be found in char-devices branch

PR is ready at #42


v2.1.0 release failed to build in VS2019

After getting the latest release 2.1.0 from GitHub, the BlazorHero CleanArchitecture failed to build in VS2019 on Windows 10. I'm getting below error when building the BlazorHero.CleanArchitecture.Application Project. Anyone know how to fix this issue? Thanks.

where is your project folder located? can I see the entire path?

For your understanding, My test application (Blazor Hero) is here - E:\NUGET\BH\SomeApplication. It builds fine without any exceptions. I suspect that your path where the project lives is too long and windows has a hard time with it. You should probably move your project to a simpler PATH.

Hi @iammukeshm,
My BlazorHero Application project path is in: C:\Users\admin\Documents\Source\BlazorHero\CleanArchitecture\BlazorHero.CleanArchitecture.Application
Anyway to overcome this long path restriction instead of moving it to C: drive root folder? Thanks!

Since this is a more Windows Specific configuration, I wouldn't actually recommend doing it. There is a key in Registry which you might have to enable. Check this ticket - https://stackoverflow.com/questions/8745215/best-way-to-resolve-file-path-too-long-exception

It is believed to move your project to simplest path as possible. It is just not necessary to move your project into very long folder stacks.


Unexpected large minting

In Septembers minting event, on farmer noticed two transactions from the minting wallet much larger than anything they would expect for their nodes:
Operation:<PHONE_NUMBER>70202113
Amount: 61,570.9875 TFT
Receiving wallet: GDGTVMSSE6S6Q3PN6KF4Q4MLLDV4WWNLJM5D34V5DQ75HCMLXGBQNO4Y
Memo (Hex HASH): fc8c173c46e7752f9d13f635335e2cf643aa75214baefccc9dc919660c6b9b6d
Operation:<PHONE_NUMBER>85488641
Amount: 10,140.6125 TFT
Receiving wallet: GDGTVMSSE6S6Q3PN6KF4Q4MLLDV4WWNLJM5D34V5DQ75HCMLXGBQNO4Y
Memo (Hex HASH): 6dba4071b6251833301cdf74fc3d026045643a257743715fc7e96862d1624625
The attached hashes don't return any results from the minting receipt visualizer. This farmer operates nodes on testnet, and these transactions seem to be linked to two testnet nodes with gateway configurations (nodes 219 and 222).

Yes, this was checked yesterday morning. The 2 payouts are related to test net nodes on v3. The receipt visualizer currently does not support test net receipts.
Regarding the actual payout, this is correct as far as the minting is concerned. With the introduction of gold farming, the minting code was updated to use farming policies from the chain itself, rather than hard coded values. As it turns out, on test net, there is a farming policy with ridiculous rewards for IP usage and NU. Iirc, values in that policy are something north of 500 USD per hour of public IP usage and more than 2 USD per NU (GB of public traffic). These 2 nodes had 10 hours of IP usage combined and about 2NU, hence the huge payout.
TL;DR: There is a broken farming certification on testnet which needs to be removed and unlinked from these nodes, or needs to be adjusted. The minting did what is was supposed to do.


Remove salvage mobs being able to be ghost roles.

About the PR

Following maintainer consensus, this is being removed for a number of reasons:

players abusing the role to go to the station and cause chaos
it's not particularly fun for players
other reasons that i forgot

Changelog
:cl: Ubaser

remove: Salvage mobs can no longer be ghost roles.

When we bump expeditions / vgroid can maybe come back but for wrecks it's too stinky.

Aw but it was fun giving salvage a challenge

And so, after this we will still have the ghost roles except mice from the spawner every 10 minutes?


[New Builder Suite] - Date & Time Suite

This PR introduces an entire "suite" of new builders - Date & Time Suite which consists of the following builders:

Calendar - build customizable, accessible, and powerful calendar components
Date Field - an enhanced alternative to the native <input type="date" /> with an emphasis on localization & accessibility
Date Picker - a combination of the Calendar, Date Field, &  Popover builders
Range Calendar - build a calendar with range selection. Extends the power & functionality of the Calendar
Date Range Field - an enhanced alternative to using two native <input type="date" /> inputs for selecting start and end dates, with an emphasis on localization & accessibility
Date Range Picker - a combination of the Range Calendar, Date Range Field, & Popover builders

Please note, this is a large introduction into the library, and although we have tests, we can't possibly catch all the bugs, so if you find any, please create an issue and we'll address it.
Additionally, if anyone would like to contribute to the documentation, we always welcome assistance there!
Todos:

[ ] Cleanup/finalize docs
[ ] Mobile calendar styles
[x] API References 😭
[ ] Make tests pass regardless of the GH runner's timezone
[ ] Add events/ActionReturn types

Should be good to start review @TGlide, the remainder of the due outs are more style-related.

@huntabyte how much time will it take to release this builder one?? I'm looking for Date and Time component.

@huntabyte how much time will it take to release this builder one?? I'm looking for Date and Time component.


"Connect to" picker is empty when quickly clicked after window opening

Testing #186222

A second click reveals:

We wait till extension gallery service availability before populating this list. Ill add a context key to hide the start entry till all items are populated.

I confirmed that when the window first opens, the "Connect to..." start entry isn't there, but after it appears, clicking on it reveals all the options.


additionalVulnerabilityReportFields is not working as expected.

What steps did you take and what happened:
I would like to extract additional fields like 'Description' and 'Title' from Trivy Operator reports or Prometheus metrics. I followed the documentation, added the extra fields in the values.yaml file, and then deployed the operator.
In values.yaml, I have made additions to this key "additionalVulnerabilityReportFields"
additionalVulnerabilityReportFields: "Description,Links,CVSS,Targets"
What did you expect to happen:
I'm expecting trivy to show the additional fields which are added in values.yaml. Those should be available as labels in prometheus metrics.
Environment:

Trivy-Operator version (use trivy-operator version): 0.24.0
Kubernetes version (use kubectl version): 1.30

Not sure if you fixed this already, but I set the metricsVulnIdEnabled under the operator setting to true:
operator:
  metricsVulnIdEnabled: "true"
I didn't use the additionalVulnerabilityReportFields, because it did not work for some reason.


Create Inventory Model

User Story
For a company handling different booth build orders I need a database that will store an object called "Inventory" that will contain information on inventory stock.
Acceptance Criteria
When I  create a model called Inventory  it should contain the following information:

Item ID
Item Description/ Name
Current Cost
Available Quantity/Total Quantity.
List of Active Projects links to inventory

This has been completed


config: PR#23 – Update Gemfile(jekyll, minima)

Dependency Error: Yikes! It looks like you don't have jekyll-readme-index or one of its dependencies installed. In order to use Jekyll as currently configured, you'll need to install this gem. The full error message from Ruby is: 'cannot load such file -- jekyll-readme-index' If you run into trouble, you can find helpful resources at https://jekyllrb.com/help/!


Add SDTM TS and M11 parameters to population and indication classes [API]

See feature ticket #153:
Add flag to PopulationDefinition. This will be inherited by studyCohort and StudyDesignPopulation.
includesHealthySubjects
Add flag to indication: isRareDisease -> boolean

It took me a few minutes to work out what was being asked for. Just for clarity @BSnoeijerCD @EMuhlbradt @czwickl I think the changes are:

Class PopulationDefinition: Add attribute 'includesHealthySubjects' of type Boolean
Class indication: Add attribute 'isRareDisease' of type Boolean

NOTE: Class PopulationDefinition is inherited by studyCohort and StudyDesignPopulation


fix strum animations for different playbacks

this just fixes the weird animation reset times the strums can have for botplay/opponent when the playback rate is different, bit like how it was in the chart editor
hopefully it doesn't break anything

if you were to do this, why not just put the playbackRate division in strumPlayAnim?
like:


Error: Could not spawn command

Issue
Getting below error while running rotz clone on a Ubuntu system. I tried both with and without creating the .dotfiles directory, (based on the error), however still getting the same error. Am I missing anything ?

Solved: Issue was git was not properly installed.