Move misplaced mk_restarts directory; clean up duplication of ReadCNTilFile(), ReadTileFile_RealLatLon() etc,

Issue #225 and pull request #224 addressed some cleanup of routines in the "mk_restarts" directory, including a renaming of subroutine ReadCNTilFile() to ReadTileFile_RealLatLon() for clarity.
This issue assumes that pull request #224 has been merged.
There are two additional concerns:

A few subroutines, incl. to_radian(), haversine() and ReadCNTilFile() a.k.a. ReadTileFile_RealLatLon()
appear to be duplicated in mk_restarts/getids.F90 and GEOSsurface_GridComp/Shared/Raster/comp_CATCHCN_AlbScale_parameters.F90

Subroutine "to_radian()" should probably be eliminated in favor of the corresponding MAPL routine.

It is not clear why "mk_restarts" lives in GEOScatch_GridCom/ when it addresses re-tiling for other (non-catch) GridComps.  My feeling is that some reorganization would be needed.

@sdrabenh suggest that @tclune comment on this on behalf of the SI team to see what should be done.

More duplication:
Subroutine calc_soil_moist() is defined in mk_restarts/Scale_Catch.F90 and also in mk_restarts/Scale_CatchCN.F90, both of which duplicate the functionality of subroutine catch_calc_soilmoist() in GEOSland_GridComp/Shared/lsm_routines.F90
(Note also subroutines partition() and rzequil() in Scale_Catch[CN].F90.)
Re. item 3 above (code reorganization):
E.g., subroutine Scale_CatchCN.F90 is in the GEOScatch_GridComp/ subdirectory but should probably be in the CatchCN Grid Comp dir.
@gmao-jkolassa @biljanaorescanin @rdkoster @weiyuan-jiang

I am happy to do clean up Scale_Catch.F90 and Scale_CatchCN.F90, but I think it makes sense to first move those routines to their final locations, just so that dependencies can be established correctly.
I had been discussing with Rolf via email about possibly creating a mk_restarts directory in GEOScatchCN_GridComp and I have included his response below:

Maybe it makes more sense to keep "mk_restarts" intact as a single dir but place it at the level of the Surface GridComp?  Or, if we were to go with separate "mk_restarts" directories in the Catch, CatchCN, etc GridComps, any files or subroutines that are common to more than one GridComp would need to be moved to a higher-level shared directory.

Are there any preferences or advantages to the proposed locations for the mk_restarts directory? Once we have agreed on a location, would you mind actually moving the files/directories @weiyuan-jiang ?

Just a short summary of email communication we had so far on move of  the mk_restarts directory to the GMAO_Shared.
The main issue we see is this from the mk_restarts CMakeLists.txt:
foreach (src ${exe_srcs})
string (REGEX REPLACE ".F90" "" exe ${src})
ecbuild_add_executable (
TARGET ${exe}
SOURCES ${src}
LIBS MAPL gftl-shared GEOS_SurfaceShared GEOSroute_GridComp GEOS_LandShared ${this})
endforeach ()
If mk_restarts is moved to GMAO_Shared where we wanted, agreed at Model Development meeting to move, then we can have a circular dependency. We know that parts of GEOSgcm_GridComp depends on GMAO_Shared, but now GMAO_Shared would depend on GEOSgcm_GridComp. Nothing in GMAO_Shared depends on GEOSgcm_GridComp at the moment, as far as we know.
We also plan to wait on current PR under GEOSgcm_GridComp to be resolved https://github.com/GEOS-ESM/GEOSgcm_GridComp/pull/368 to proceed with this move since that PR will add one more restart for land CatchCN CLM 4.5 to this directory.
From other communication with @sanAkel  they wish to move their ocean/sea ice stuff off of mk_restarts in any case.
Here is the list:

SaltImpConverter.F90
SaltIntSplitter.F90
cv_SaltRestart.F90
mk_CICERestart.F90
mk_RouteRestarts.F90
And the following 1-4 are for ocean ("salt") and sea-ice  ("CICE") and have nothing to do with land.
mk_RouteRestarts.F90 is new and still under development to my understanding.

I think this is the list of all interested or involved sides I apologize if I missed someone:
@weiyuan-jiang @gmao-rreichle @gmao-jkolassa @sanAkel @mathomp4 @smahanam

Just a short summary of email communication we had so far on move of  the mk_restarts directory to the GMAO_Shared.
The main issue we see is this from the mk_restarts CMakeLists.txt:
foreach (src ${exe_srcs})
string (REGEX REPLACE ".F90" "" exe ${src})
ecbuild_add_executable (
TARGET ${exe}
SOURCES ${src}
LIBS MAPL gftl-shared GEOS_SurfaceShared GEOSroute_GridComp GEOS_LandShared ${this})
endforeach ()
If mk_restarts is moved to GMAO_Shared where we wanted, agreed at Model Development meeting to move, then we can have a circular dependency. We know that parts of GEOSgcm_GridComp depends on GMAO_Shared, but now GMAO_Shared would depend on GEOSgcm_GridComp. Nothing in GMAO_Shared depends on GEOSgcm_GridComp at the moment, as far as we know.
We also plan to wait on current PR under GEOSgcm_GridComp to be resolved https://github.com/GEOS-ESM/GEOSgcm_GridComp/pull/368 to proceed with this move since that PR will add one more restart for land CatchCN CLM 4.5 to this directory.
From other communication with @sanAkel  they wish to move their ocean/sea ice stuff off of mk_restarts in any case.
Here is the list:

SaltImpConverter.F90
SaltIntSplitter.F90
cv_SaltRestart.F90
mk_CICERestart.F90
mk_RouteRestarts.F90
And the following 1-4 are for ocean ("salt") and sea-ice  ("CICE") and have nothing to do with land.
mk_RouteRestarts.F90 is new and still under development to my understanding.

I think this is the list of all interested or involved sides I apologize if I missed someone:
@weiyuan-jiang @gmao-rreichle @gmao-jkolassa @sanAkel @mathomp4 @smahanam

Quick correction:  "mk_RouteRestarts.F90" is part of the land -- it addresses the runoff routing (streamflow) module that is under development.

Quick correction:  "mk_RouteRestarts.F90" is part of the land -- it addresses the runoff routing (streamflow) module that is under development.

@yvikhlya @zhaobin74 and myself will work to get
1. SaltImpConverter.F90
2. SaltIntSplitter.F90
3. cv_SaltRestart.F90
4. mk_CICERestart.F90

out of mk_restarts FYI: @GEOS-ESM/cmake-team

@yvikhlya @zhaobin74 and myself will work to get
1. SaltImpConverter.F90
2. SaltIntSplitter.F90
3. cv_SaltRestart.F90
4. mk_CICERestart.F90

out of mk_restarts FYI: @GEOS-ESM/cmake-team

@yvikhlya @zhaobin74 and myself will work to get
1. SaltImpConverter.F90
2. SaltIntSplitter.F90
3. cv_SaltRestart.F90
4. mk_CICERestart.F90

out of mk_restarts FYI: @GEOS-ESM/cmake-team

But these are actually utils to create surface restarts. Should the whole mk_restarts be moved to GEOSsurface_GridComp/Shared instead? @tclune @mathomp4

@yvikhlya @zhaobin74 and myself will work to get
1. SaltImpConverter.F90
2. SaltIntSplitter.F90
3. cv_SaltRestart.F90
4. mk_CICERestart.F90

out of mk_restarts FYI: @GEOS-ESM/cmake-team

But these are actually utils to create surface restarts. Should the whole mk_restarts be moved to GEOSsurface_GridComp/Shared instead? @tclune @mathomp4

@yvikhlya please ⏫ https://github.com/GEOS-ESM/GEOSgcm_GridComp/issues/232#issuecomment-768495703

@yvikhlya please ⏫ https://github.com/GEOS-ESM/GEOSgcm_GridComp/issues/232#issuecomment-768495703

But these are actually utils to create surface restarts. Should the whole mk_restarts be moved to GEOSsurface_GridComp/Shared instead? @tclune @mathomp4

Yuri has a valid point.
I am not recommending anything over a something that has been discussed at length and decided already.
This is merely my observation too, GEOSsurface_GridComp/Shared is not a bad place for this mk_restarts directory. 'Raster" can be renamed to "mk_surface_bcs" or something.

But these are actually utils to create surface restarts. Should the whole mk_restarts be moved to GEOSsurface_GridComp/Shared instead? @tclune @mathomp4

Yuri has a valid point.
I am not recommending anything over a something that has been discussed at length and decided already.
This is merely my observation too, GEOSsurface_GridComp/Shared is not a bad place for this mk_restarts directory. 'Raster" can be renamed to "mk_surface_bcs" or something.

I agree  with Yury. The ocean/sea ice regridding should stay with the others.

I agree  with Yury. The ocean/sea ice regridding should stay with the others.

@yvikhlya please ⏫ #232 (comment)

Biljana is talking about difficulties to move mk_restarts to GMAO_Shared, but GEOSsurf_GridComp has its own Shared directory and as far as I remember everything under mk_restarts is related to surface restarts.

@yvikhlya please ⏫ #232 (comment)

Biljana is talking about difficulties to move mk_restarts to GMAO_Shared, but GEOSsurf_GridComp has its own Shared directory and as far as I remember everything under mk_restarts is related to surface restarts.

@smahanam nothing was decided. We proposed to re-organize so that individual restart routines are in more appropriate location( more intuitive to locate for user). Since we all agree on that currently mk_restarts is located in GEOSCatch_GridComp but contains many routines that are not related to Catchment.
We proposed these two options at model development meeting:

Keep mk_restarts directory intact as it is, but move to a higher level folder, e.g. GEOSSurface_GridComp
Create mk_restarts directory in each GridComp subdirectory and move restarts routines to appropriate directory.
I think we hoped people would like option 2. , like it would work for everyone but @sanAkel suggested moving it to the GEOS_Util. Which turned out to be in circular dependency then.

@smahanam nothing was decided. We proposed to re-organize so that individual restart routines are in more appropriate location( more intuitive to locate for user). Since we all agree on that currently mk_restarts is located in GEOSCatch_GridComp but contains many routines that are not related to Catchment.
We proposed these two options at model development meeting:

Keep mk_restarts directory intact as it is, but move to a higher level folder, e.g. GEOSSurface_GridComp
Create mk_restarts directory in each GridComp subdirectory and move restarts routines to appropriate directory.
I think we hoped people would like option 2. , like it would work for everyone but @sanAkel suggested moving it to the GEOS_Util. Which turned out to be in circular dependency then.

No, GEOS_Util is not good location for mk_restarts.

No, GEOS_Util is not good location for mk_restarts.

@yvikhlya @smahanam  It sounds like you both agree on moving it as it is but to the https://github.com/GEOS-ESM/GEOSgcm_GridComp/tree/develop/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/Shared?
@gmao-rreichle @weiyuan-jiang

@yvikhlya @smahanam  It sounds like you both agree on moving it as it is but to the https://github.com/GEOS-ESM/GEOSgcm_GridComp/tree/develop/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/Shared?
@gmao-rreichle @weiyuan-jiang

@yvikhlya @smahanam It sounds like you both agree on moving it as it is but to the https://github.com/GEOS-ESM/GEOSgcm_GridComp/tree/develop/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/Shared?
@gmao-rreichle @weiyuan-jiang

Yes, I would suggest to do so if nobody objects.

@yvikhlya @smahanam It sounds like you both agree on moving it as it is but to the https://github.com/GEOS-ESM/GEOSgcm_GridComp/tree/develop/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/Shared?
@gmao-rreichle @weiyuan-jiang

Yes, I would suggest to do so if nobody objects.

I agree with Yury. The ocean/sea ice regridding should stay with the others.

why should they

should stay with the others.
?

I agree with Yury. The ocean/sea ice regridding should stay with the others.

why should they

should stay with the others.
?

@smahanam nothing was decided. We proposed to re-organize so that individual restart routines are in more appropriate location( more intuitive to locate for user). Since we all agree that currently mk_restarts is located in GEOSCatch_GridComp but contains many routines that are not related to Catchment.
We proposed these two options at model development meeting:

Keep mk_restarts directory intact as it is, but move to a higher level folder, e.g. GEOSSurface_GridComp
Create mk_restarts directory in each GridComp subdirectory and move restarts routines to appropriate directory.
I think we hoped people would like option 2. , like it would work for everyone but @sanAkel suggested moving it to the GEOS_Util. Which turned out to be in circular dependency then.

@biljanaorescanin

but @sanAkel suggested moving it to the GEOS_Util. Which turned out to be in circular dependency then.

This suggestion is outdated! Please see https://github.com/GEOS-ESM/GEOSgcm_App/discussions/193 instead.

@smahanam nothing was decided. We proposed to re-organize so that individual restart routines are in more appropriate location( more intuitive to locate for user). Since we all agree that currently mk_restarts is located in GEOSCatch_GridComp but contains many routines that are not related to Catchment.
We proposed these two options at model development meeting:

Keep mk_restarts directory intact as it is, but move to a higher level folder, e.g. GEOSSurface_GridComp
Create mk_restarts directory in each GridComp subdirectory and move restarts routines to appropriate directory.
I think we hoped people would like option 2. , like it would work for everyone but @sanAkel suggested moving it to the GEOS_Util. Which turned out to be in circular dependency then.

@biljanaorescanin

but @sanAkel suggested moving it to the GEOS_Util. Which turned out to be in circular dependency then.

This suggestion is outdated! Please see https://github.com/GEOS-ESM/GEOSgcm_App/discussions/193 instead.

I agree with Yury. The ocean/sea ice regridding should stay with the others.

@zhaobin74 why should they

should stay with the others.
?

because they are all children of surface

I agree with Yury. The ocean/sea ice regridding should stay with the others.

@zhaobin74 why should they

should stay with the others.
?

because they are all children of surface

I agree with Yury. The ocean/sea ice regridding should stay with the others.

@zhaobin74 why should they

should stay with the others.
?

because they are all children of surface

where is the explicit connectivity? line numbers?

I agree with Yury. The ocean/sea ice regridding should stay with the others.

@zhaobin74 why should they

should stay with the others.
?

because they are all children of surface

where is the explicit connectivity? line numbers?

I agree with Yury. The ocean/sea ice regridding should stay with the others.

@zhaobin74 why should they

should stay with the others.
?

because they are all children of surface

where is the explicit connectivity? line numbers?

These mk_*Restarts are not well separated. For example mk_LakeLaniceSaltRestarts makes lake, landice and saltwater restarts, so where would we put it?

I agree with Yury. The ocean/sea ice regridding should stay with the others.

@zhaobin74 why should they

should stay with the others.
?

because they are all children of surface

where is the explicit connectivity? line numbers?

These mk_*Restarts are not well separated. For example mk_LakeLaniceSaltRestarts makes lake, landice and saltwater restarts, so where would we put it?

@yvikhlya @smahanam , In a GridComp, usually the Shared directory is built first, then the children GridComp and the siblings which depends on the shared directory.  If you put all the mk_restart in Shared directories, there would be dependency problems. We either move all mk_restarts under SurfaceGridComp that depends on all the children and siblings or create a mk_restart directory under each grid comp.

@yvikhlya @smahanam , In a GridComp, usually the Shared directory is built first, then the children GridComp and the siblings which depends on the shared directory.  If you put all the mk_restart in Shared directories, there would be dependency problems. We either move all mk_restarts under SurfaceGridComp that depends on all the children and siblings or create a mk_restart directory under each grid comp.

create a mk_restart directory under each grid comp.

sounds better.

create a mk_restart directory under each grid comp.

sounds better.

Then where mk_LakeLandiceSaltRestarts would go?

Then where mk_LakeLandiceSaltRestarts would go?

under "surface" following @weiyuan-jiang idea

under each grid comp

under "surface" following @weiyuan-jiang idea

under each grid comp

Why does  landlice, lake and saltwater need  to have duplicated utilities if one works for them all?

Why does  landlice, lake and saltwater need  to have duplicated utilities if one works for them all?

I saw the dependencies in mk_restarts/CMakelists.txt here, but that may be false dependency. let me verify this first
foreach (src ${exe_srcs})
string (REGEX REPLACE ".F90" "" exe ${src})
ecbuild_add_executable (
TARGET ${exe}
SOURCES ${src}
LIBS MAPL gftl-shared GEOS_SurfaceShared GEOSroute_GridComp GEOS_LandShared ${this})
endforeach ()

I saw the dependencies in mk_restarts/CMakelists.txt here, but that may be false dependency. let me verify this first
foreach (src ${exe_srcs})
string (REGEX REPLACE ".F90" "" exe ${src})
ecbuild_add_executable (
TARGET ${exe}
SOURCES ${src}
LIBS MAPL gftl-shared GEOS_SurfaceShared GEOSroute_GridComp GEOS_LandShared ${this})
endforeach ()

I don't get how simply moving mk_restarts to GEOSsurf_GridComp/Shared would create dependency problems? Right now it is in a completely weird location and we don't have dependency problems.
Creating separate mk_restarts under each surface child is a possible solution, but it looks to me a little more complex than just moving the whole thing.

I don't get how simply moving mk_restarts to GEOSsurf_GridComp/Shared would create dependency problems? Right now it is in a completely weird location and we don't have dependency problems.
Creating separate mk_restarts under each surface child is a possible solution, but it looks to me a little more complex than just moving the whole thing.

We do have some dependency problems. For example, the file Scale_Catch.F90 depends on land_shared.  We cannot just move this program to GEOSsurf_GridComp/Shared  because GEOSland_GridComp/Shared depends on the upper level's Shared

We do have some dependency problems. For example, the file Scale_Catch.F90 depends on land_shared.  We cannot just move this program to GEOSsurf_GridComp/Shared  because GEOSland_GridComp/Shared depends on the upper level's Shared

But we can move everything to new GEOSsurf_GridComp/mk_restarts . That will be a very simple solution @yvikhlya

But we can move everything to new GEOSsurf_GridComp/mk_restarts . That will be a very simple solution @yvikhlya

I understand Weiyuan's difficulty, but we can ask Tom - he might have a suggestion. Maybe, 'Shared' is a wrong name for the content then, these codes are actually "Application" category - they don't share modules. They use others modules.
I just saw Weiyuan's message (but please put Raster and mk_restarts together inside a new directory)

I vote to keep all land, landice, sea, seaice together in one directory.
the land programs wish to stay inside GEOSgcm_GridComp repo for 2 reasons:

They have use statements to science modules and libraries because those codes are part of science.
Some science features sometimes  need upgrades to restart files, surface bcs files and scripts, in addition to the modifications that we make to to grid components, . It makes developer's job easy when everything in the same repo. For e.g. I have 3 or 4 active branches in GESOgcm_GridComp, I work in tandem in both in GEOS**_GridComp, mk_restarts in the same repo.

another question, shouldn't GEOS_Util/post/regrid.pl join the same directory should all stay together?

I understand Weiyuan's difficulty, but we can ask Tom - he might have a suggestion. Maybe, 'Shared' is a wrong name for the content then, these codes are actually "Application" category - they don't share modules. They use others modules.
I just saw Weiyuan's message (but please put Raster and mk_restarts together inside a new directory)

I vote to keep all land, landice, sea, seaice together in one directory.
the land programs wish to stay inside GEOSgcm_GridComp repo for 2 reasons:

They have use statements to science modules and libraries because those codes are part of science.
Some science features sometimes  need upgrades to restart files, surface bcs files and scripts, in addition to the modifications that we make to to grid components, . It makes developer's job easy when everything in the same repo. For e.g. I have 3 or 4 active branches in GESOgcm_GridComp, I work in tandem in both in GEOS**_GridComp, mk_restarts in the same repo.

another question, shouldn't GEOS_Util/post/regrid.pl join the same directory should all stay together?

But we can move everything to new GEOSsurf_GridComp/mk_restarts . That will be a very simple solution @yvikhlya

I would support this.

But we can move everything to new GEOSsurf_GridComp/mk_restarts . That will be a very simple solution @yvikhlya

I would support this.

@yvikhlya @smahanam,  We are closer and closer. How about we create GEOSsurf_GridComp/Utils which includes two sub directrories mk_restarts and Rasters?  I think we can move some files from Rasters to GEOSsurf_GridComp/Shared . The file GEOS_Util/post/regrid.pl can be moved to GEOSsurf_GridComp/Utils too.

@yvikhlya @smahanam,  We are closer and closer. How about we create GEOSsurf_GridComp/Utils which includes two sub directrories mk_restarts and Rasters?  I think we can move some files from Rasters to GEOSsurf_GridComp/Shared . The file GEOS_Util/post/regrid.pl can be moved to GEOSsurf_GridComp/Utils too.

Weiyuan, sounds good to me - I don't involve in that kind of decisions, though.

@zhaobin74 why should they

Land wanst to keep an eye on other groups, for e.g., landice got caught red-handed once while encroaching on our prime real estate - land pixels. Somebody said AOGCM is encroaching our beaches. Those are the real questions that trouble us - just kidding.

Weiyuan, sounds good to me - I don't involve in that kind of decisions, though.

@zhaobin74 why should they

Land wanst to keep an eye on other groups, for e.g., landice got caught red-handed once while encroaching on our prime real estate - land pixels. Somebody said AOGCM is encroaching our beaches. Those are the real questions that trouble us - just kidding.

@yvikhlya @smahanam, We are closer and closer. How about we create GEOSsurf_GridComp/Utils which includes two sub directrories mk_restarts and Rasters? I think we can move some files from Rasters to GEOSsurf_GridComp/Shared . The file GEOS_Util/post/regrid.pl can be moved to GEOSsurf_GridComp/Utils too.

regrid.pl regrids all restarts, not just surface actually. So, it probably belongs to some higher level component.

@yvikhlya @smahanam, We are closer and closer. How about we create GEOSsurf_GridComp/Utils which includes two sub directrories mk_restarts and Rasters? I think we can move some files from Rasters to GEOSsurf_GridComp/Shared . The file GEOS_Util/post/regrid.pl can be moved to GEOSsurf_GridComp/Utils too.

regrid.pl regrids all restarts, not just surface actually. So, it probably belongs to some higher level component.

if you want to 👀 you can make yourself be included in "code owners" that's a request to @mathomp4

if you want to 👀 you can make yourself be included in "code owners" that's a request to @mathomp4

@yvikhlya @smahanam, We are closer and closer. How about we create GEOSsurf_GridComp/Utils which includes two sub directrories mk_restarts and Rasters? I think we can move some files from Rasters to GEOSsurf_GridComp/Shared . The file GEOS_Util/post/regrid.pl can be moved to GEOSsurf_GridComp/Utils too.

regrid.pl regrids all restarts, not just surface actually. So, it probably belongs to some higher level component.

SURE ! If you say so,

@yvikhlya @smahanam, We are closer and closer. How about we create GEOSsurf_GridComp/Utils which includes two sub directrories mk_restarts and Rasters? I think we can move some files from Rasters to GEOSsurf_GridComp/Shared . The file GEOS_Util/post/regrid.pl can be moved to GEOSsurf_GridComp/Utils too.

regrid.pl regrids all restarts, not just surface actually. So, it probably belongs to some higher level component.

SURE ! If you say so,

But we can move everything to new GEOSsurf_GridComp/mk_restarts . That will be a very simple solution @yvikhlya

I would support this.

second this

But we can move everything to new GEOSsurf_GridComp/mk_restarts . That will be a very simple solution @yvikhlya

I would support this.

second this

OK. Let me restructure and move around the mk_restarts first.  Not much work, so even if it is not approved, it is fine :-)

OK. Let me restructure and move around the mk_restarts first.  Not much work, so even if it is not approved, it is fine :-)

OK. Let me restructure and move around the mk_restarts first. Not much work, so even if it is not approved, it is fine :-)

@tclune does this sound good to you?

OK. Let me restructure and move around the mk_restarts first. Not much work, so even if it is not approved, it is fine :-)

@tclune does this sound good to you?

Ok, folks, I have pushed a branch feature/wjiang/mk_restarts_re_org  to github that relocates mk_restarts and Raster.  Basically I add a sub-directory Utils under GEOSsurface_GridComp, and move Raster and mk_restarts under Utils.  As Yury suggested, we are better not to move regrid.pl at this point.  If you don't have objection,  I will talk to @tclune  and @mathomp4 about this change.

Ok, folks, I have pushed a branch feature/wjiang/mk_restarts_re_org  to github that relocates mk_restarts and Raster.  Basically I add a sub-directory Utils under GEOSsurface_GridComp, and move Raster and mk_restarts under Utils.  As Yury suggested, we are better not to move regrid.pl at this point.  If you don't have objection,  I will talk to @tclune  and @mathomp4 about this change.

Many thanks to everyone for the inspired discussion, and to @weiyuan-jiang for implementing a workable option.
Moving the whole package to Surface is far better than keeping it in Catch.  Putting the package into a new Surface/util directory should get us around the circular dependency that we would otherwise have with mk_Catch[CN]restarts in Surface/Shared or GEOS_Util.  (Sorry, Catch restarts are a pain...)
To move things to GEOS_Util is theoretically possible but it would require meticulously dissecting mk_Catch[CN]restarts, which would be considerably harder and for which we probably don't have the resources.  It would also be far riskier because there's no routine zero-diff test for mkRestarts.

Many thanks to everyone for the inspired discussion, and to @weiyuan-jiang for implementing a workable option.
Moving the whole package to Surface is far better than keeping it in Catch.  Putting the package into a new Surface/util directory should get us around the circular dependency that we would otherwise have with mk_Catch[CN]restarts in Surface/Shared or GEOS_Util.  (Sorry, Catch restarts are a pain...)
To move things to GEOS_Util is theoretically possible but it would require meticulously dissecting mk_Catch[CN]restarts, which would be considerably harder and for which we probably don't have the resources.  It would also be far riskier because there's no routine zero-diff test for mkRestarts.

I don't know what the right near-term location should be for the mk_restarts.   One thing I am considering in the longer term, is that as we establish a data repository (Git LSF) for model inputs, maybe these programs belong in that repository?      I'm interested in opinions.
The pro's of putting these in the data repository is that it makes it clear how the input data are created and what to change to generate new data.    OTOH,  these programs presumably evolve with the components that they support, which would argue to co-locate the program with the grid comp.
If the programs were truly standalone, I'd  definitely push for the former, but to the extent they need to link to GEOS libraries, the latter has appeal.
Am interested in all opinions.   Soon I'll be drafting a proposal for the data repository and give senior staff the opportunity to chime in.

I don't know what the right near-term location should be for the mk_restarts.   One thing I am considering in the longer term, is that as we establish a data repository (Git LSF) for model inputs, maybe these programs belong in that repository?      I'm interested in opinions.
The pro's of putting these in the data repository is that it makes it clear how the input data are created and what to change to generate new data.    OTOH,  these programs presumably evolve with the components that they support, which would argue to co-locate the program with the grid comp.
If the programs were truly standalone, I'd  definitely push for the former, but to the extent they need to link to GEOS libraries, the latter has appeal.
Am interested in all opinions.   Soon I'll be drafting a proposal for the data repository and give senior staff the opportunity to chime in.

@tclune : Good to hear about the plans for a data repository.  That's attractive for many reasons, but it won't solve the fact that at least for Catch[CN] restarts, the bulk of the code that processes restarts needs to stay within the GCM GridComp repository.  For Catch[CN], creating restarts involves a good helping of science routines that live in the GCM GridComp, and it doesn't make sense to duplicate these in a different repository.  So we could at best move some top-level scripts into the data repository, but not the whole package.  What's more, the current mkRestarts package cannot be split easily into elements that remain in the GCM GridComp and elements that can be moved outside the repository.  Put differently, moving mkRestarts to a data repository is no easier than moving it to GEOS_Util.  We may find that Surface/util is the only realistic option for mkRestarts.
I don't know if the "Raster" s/w to create boundary conditions would split more easily.  @smahanam should be able to comment on this.
For now, I think we should go ahead with Weiyuan's branch that moves mkRestarts and Raster to Surface/util.  That would definitely be an improvement over the status quo.  This doesn't mean we can't discuss further modifications, but those would take more time and effort.

@tclune : Good to hear about the plans for a data repository.  That's attractive for many reasons, but it won't solve the fact that at least for Catch[CN] restarts, the bulk of the code that processes restarts needs to stay within the GCM GridComp repository.  For Catch[CN], creating restarts involves a good helping of science routines that live in the GCM GridComp, and it doesn't make sense to duplicate these in a different repository.  So we could at best move some top-level scripts into the data repository, but not the whole package.  What's more, the current mkRestarts package cannot be split easily into elements that remain in the GCM GridComp and elements that can be moved outside the repository.  Put differently, moving mkRestarts to a data repository is no easier than moving it to GEOS_Util.  We may find that Surface/util is the only realistic option for mkRestarts.
I don't know if the "Raster" s/w to create boundary conditions would split more easily.  @smahanam should be able to comment on this.
For now, I think we should go ahead with Weiyuan's branch that moves mkRestarts and Raster to Surface/util.  That would definitely be an improvement over the status quo.  This doesn't mean we can't discuss further modifications, but those would take more time and effort.

@tclune Managing versions of data files with LFS would be a great option imo. As for programs which generate data files, I am not sure where it is better to keep them. I guess, it will become more clear when LFS option is available. To generate a coupled model grid, we use a whole bunch of programs. Some of these programs are stand alone, but there are some which depend on GEOS libraries.

@tclune Managing versions of data files with LFS would be a great option imo. As for programs which generate data files, I am not sure where it is better to keep them. I guess, it will become more clear when LFS option is available. To generate a coupled model grid, we use a whole bunch of programs. Some of these programs are stand alone, but there are some which depend on GEOS libraries.

To generate a coupled model grid, we use a whole bunch of programs. Some of these programs are stand alone, but there are some which depend on GEOS libraries.

@yvikhlya would please list them  and their exact dependencies. Thanks! Please also coordinate with @zhaobin74 to include sea ice and of course don't forget the (river) routing.
@lcandre2 please add to this list your glacier routing routines as well. Thanks!
@tclune everything from the ocean model: MOM6 is entire outside of GEOS and is standalone, I believe same applies to CICE6, our future sea ice module. AFAIK what @yvikhlya is referring to ⏫ is all for coupling with GEOS. The reason for including this aspect (of connections to GEOS) is with the thought of having clean data drivers for our external components, e.g., MOM6, CICE6, etc.

To generate a coupled model grid, we use a whole bunch of programs. Some of these programs are stand alone, but there are some which depend on GEOS libraries.

@yvikhlya would please list them  and their exact dependencies. Thanks! Please also coordinate with @zhaobin74 to include sea ice and of course don't forget the (river) routing.
@lcandre2 please add to this list your glacier routing routines as well. Thanks!
@tclune everything from the ocean model: MOM6 is entire outside of GEOS and is standalone, I believe same applies to CICE6, our future sea ice module. AFAIK what @yvikhlya is referring to ⏫ is all for coupling with GEOS. The reason for including this aspect (of connections to GEOS) is with the thought of having clean data drivers for our external components, e.g., MOM6, CICE6, etc.

@gmao-rreichle  I understand that these programs have dependencies on things that must live in the main software repositories.    Hence the conflicting concerns.      If moved into the data repository, there would also be a bit of CMake logic that would do the necessary linking from in installation of GEOS.     Not a great situation maybe, but also not intractable.   Certainly the solution you have chosen is fine for the reasonably foreseeable future.     I would just want to revisit once the data repo plans firm up a bit.

@gmao-rreichle  I understand that these programs have dependencies on things that must live in the main software repositories.    Hence the conflicting concerns.      If moved into the data repository, there would also be a bit of CMake logic that would do the necessary linking from in installation of GEOS.     Not a great situation maybe, but also not intractable.   Certainly the solution you have chosen is fine for the reasonably foreseeable future.     I would just want to revisit once the data repo plans firm up a bit.

would please list them and their exact dependencies

Oh, don't ask me please, because I have no idea what those dependencies are.

would please list them and their exact dependencies

Oh, don't ask me please, because I have no idea what those dependencies are.

@yvikhlya would please list them and their exact dependencies. Thanks! Please also coordinate with > @zhaobin74 to include sea ice and of course don't forget the (river) routing.

cice grid generator doesn't have any dependencies except for the baselibs.
As @sanAkel  mentioned, moving forward we 'll have a CICE6 repo which contains all the utilities for grid and restart generation. The current set of tools in mk_restart will be carried for legacy purpose.

@yvikhlya would please list them and their exact dependencies. Thanks! Please also coordinate with > @zhaobin74 to include sea ice and of course don't forget the (river) routing.

cice grid generator doesn't have any dependencies except for the baselibs.
As @sanAkel  mentioned, moving forward we 'll have a CICE6 repo which contains all the utilities for grid and restart generation. The current set of tools in mk_restart will be carried for legacy purpose.

would please list them and their exact dependencies

Oh, don't ask me please, because I have no idea what those dependencies are.

@yvikhlya since you brought up
Some of these programs are stand alone, but there are some which depend on GEOS libraries.

I assumed you are well aware of it. In any case, I feel it is not impossible to grep use module from the head of the .F90! For instance, I can list following:
https://github.com/GEOS-ESM/GEOSgcm_GridComp/blob/e0cf9fabbe700c3e23571466f444973dd6b20e4e/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/SaltImpConverter.F90#L3-L7

would please list them and their exact dependencies

Oh, don't ask me please, because I have no idea what those dependencies are.

@yvikhlya since you brought up
Some of these programs are stand alone, but there are some which depend on GEOS libraries.

I assumed you are well aware of it. In any case, I feel it is not impossible to grep use module from the head of the .F90! For instance, I can list following:
https://github.com/GEOS-ESM/GEOSgcm_GridComp/blob/e0cf9fabbe700c3e23571466f444973dd6b20e4e/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/SaltImpConverter.F90#L3-L7

I assumed you are well aware of it

No, I don't. I have no idea how the tile file, routing table and land BCs are generated.

I assumed you are well aware of it

No, I don't. I have no idea how the tile file, routing table and land BCs are generated.

OK we are talking about mk_restarts:
2 reasons why they all should stay together:

all programs use the same master_til_file reader and getids function within the same directory
In 2019, I made some changes to getids, WJ too made some changes later. We did not know what mk_ice, mk_saltice are doing, but we saw there are other programs that use the same subroutines, so we modified the calls in those programs too.
They use MAPL, in fact, they should be using more from science modules, there are many instances that are duplicates to some science routines.

Raster:
It is a bit long story, I'll get back to you all later today.
Tom, I am not going to be inconvenienced if you decide moving all these somewhere if a better long term option.  I withdraw all my comments above, if the changes are needed to handle these rapidly evolving improvements (we land added 4 programs and revamped an old one in last 4 years), I see mk_ice etc are new too.

OK we are talking about mk_restarts:
2 reasons why they all should stay together:

all programs use the same master_til_file reader and getids function within the same directory
In 2019, I made some changes to getids, WJ too made some changes later. We did not know what mk_ice, mk_saltice are doing, but we saw there are other programs that use the same subroutines, so we modified the calls in those programs too.
They use MAPL, in fact, they should be using more from science modules, there are many instances that are duplicates to some science routines.

Raster:
It is a bit long story, I'll get back to you all later today.
Tom, I am not going to be inconvenienced if you decide moving all these somewhere if a better long term option.  I withdraw all my comments above, if the changes are needed to handle these rapidly evolving improvements (we land added 4 programs and revamped an old one in last 4 years), I see mk_ice etc are new too.

OK we are talking about mk_restarts

Yes, exactly. @sanAkel we have a number of things with generating a coupled set up which don't go smoothly and need to be addressed, but let's not bring them up here. This is a subject for a separate issue (a big one).

OK we are talking about mk_restarts

Yes, exactly. @sanAkel we have a number of things with generating a coupled set up which don't go smoothly and need to be addressed, but let's not bring them up here. This is a subject for a separate issue (a big one).

OK we are talking about mk_restarts

Yes, exactly. @sanAkel we have a number of things with generating a coupled set up which don't go smoothly and need to be addressed, but let's not bring them up here. This is a subject for a separate issue (a big one).

okay I look forward to seeing that with its gorgeous details! 🌞

OK we are talking about mk_restarts

Yes, exactly. @sanAkel we have a number of things with generating a coupled set up which don't go smoothly and need to be addressed, but let's not bring them up here. This is a subject for a separate issue (a big one).

okay I look forward to seeing that with its gorgeous details! 🌞

Hi Tom,
For the proposed data repository - this is the thread you asked people for their input.
GCM reads few input files from $BCSDIR/$BCRSLV/.
I am describing the background of the files those I am involved (from the land group).
I'll use /discover/nobackup/ltakacs/bcs/Icarus-NLv3/Icarus-NLv3_MERRA-2/CF0090x6C_DE1440xPE0720/ directory as an example.
Programs in GEOSsurface_GridComp/Shared/Raster created below files and 'rst', 'til','clsm' subdirectories.
CF0090x6C_DE1440xPE0720-Pfafstetter.til
green_clim_90x540.data
lai_clim_90x540.data
lnfm_clim_90x540.data
ndvi_clim_90x540.data
nirdf_90x540.dat
vegdyn_90x540.dat
visdf_90x540.dat
irrigation_90x540.dat (future)
GCM reads above 9 files directly, while programs in mk_restarts, in addition to writing initial model state, read parameter
files in 'clsm' subdirectory and write them also into catch(cn)_internal_rst files for the GCM to read.
I am not talking about topo_DYN_ave_90x540.data, topo_GWD_var_90x540.data and topo_TRB_var_90x540.data - I think it is Ben who created those.
We use various global data sets to derive those parameter files. As an example, here are the steps we follow to produce
lai_clim_90x540.data climatology file from MODIS MCD15A2H v6 LAI data.
STEP 1: Preprocessing - This step is usually performed in local machines. I guess, this step can continue to stay in local machines.
download MODIS for all years (2002-2020, 46 days per year, ~315 files per day), reproject to lat/lon (from 500m sinosoidal to 15 arcsec),
stitch together, compute climatology on 15 arcsec pixels (46 days per year) etc. and prepare 'Analysis_Ready' global data files.
STEP 2: As of now /gpfsm/dnb42/projects/p16/ssd/land/l_data/LandBCs_files_for_mkCatchParam/V001 is our Analysis_Ready data
directory.
Programs in GEOSsurface_GridComp/Shared/Raster have data readers for files we create in Step 1. They read those files and
create fields and parameters on land tiles and write GCM input files  in /discover/nobackup/ltakacs/bcs/Icarus-NLv3/Icarus-NLv3_MERRA-2/CF0090x6C_DE1440xPE0720/
Diskspace:
Icarus-NLv3 $BCSDIR (including all grid resolutions) 1021 GB
Analysis_Ready directory: 468 GB
There is about 1.5 TB of another Analysis_Ready data set elsewhere that may be used for MERRA3. We have outsourced them to 617.0 and then to NCCS.
Above is the workflow for land files. Let me know if you have questions.
Thanks,
Sarith

Hi Tom,
For the proposed data repository - this is the thread you asked people for their input.
GCM reads few input files from $BCSDIR/$BCRSLV/.
I am describing the background of the files those I am involved (from the land group).
I'll use /discover/nobackup/ltakacs/bcs/Icarus-NLv3/Icarus-NLv3_MERRA-2/CF0090x6C_DE1440xPE0720/ directory as an example.
Programs in GEOSsurface_GridComp/Shared/Raster created below files and 'rst', 'til','clsm' subdirectories.
CF0090x6C_DE1440xPE0720-Pfafstetter.til
green_clim_90x540.data
lai_clim_90x540.data
lnfm_clim_90x540.data
ndvi_clim_90x540.data
nirdf_90x540.dat
vegdyn_90x540.dat
visdf_90x540.dat
irrigation_90x540.dat (future)
GCM reads above 9 files directly, while programs in mk_restarts, in addition to writing initial model state, read parameter
files in 'clsm' subdirectory and write them also into catch(cn)_internal_rst files for the GCM to read.
I am not talking about topo_DYN_ave_90x540.data, topo_GWD_var_90x540.data and topo_TRB_var_90x540.data - I think it is Ben who created those.
We use various global data sets to derive those parameter files. As an example, here are the steps we follow to produce
lai_clim_90x540.data climatology file from MODIS MCD15A2H v6 LAI data.
STEP 1: Preprocessing - This step is usually performed in local machines. I guess, this step can continue to stay in local machines.
download MODIS for all years (2002-2020, 46 days per year, ~315 files per day), reproject to lat/lon (from 500m sinosoidal to 15 arcsec),
stitch together, compute climatology on 15 arcsec pixels (46 days per year) etc. and prepare 'Analysis_Ready' global data files.
STEP 2: As of now /gpfsm/dnb42/projects/p16/ssd/land/l_data/LandBCs_files_for_mkCatchParam/V001 is our Analysis_Ready data
directory.
Programs in GEOSsurface_GridComp/Shared/Raster have data readers for files we create in Step 1. They read those files and
create fields and parameters on land tiles and write GCM input files  in /discover/nobackup/ltakacs/bcs/Icarus-NLv3/Icarus-NLv3_MERRA-2/CF0090x6C_DE1440xPE0720/
Diskspace:
Icarus-NLv3 $BCSDIR (including all grid resolutions) 1021 GB
Analysis_Ready directory: 468 GB
There is about 1.5 TB of another Analysis_Ready data set elsewhere that may be used for MERRA3. We have outsourced them to 617.0 and then to NCCS.
Above is the workflow for land files. Let me know if you have questions.
Thanks,
Sarith


variable values change at Runtime in  controller (IDFGH-12069)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report
Its strength but some how my esp idf variable changes automatically at runtime.
they are global variables,
(ESP IDF version 4.3.6)
ESP32 wrover IE microcontroller
This is at starting position.

This is when my one task running that uses same variable.

I double checked from everywhere that  I am not assigning zero to it.
NOTES:
I am using Bluetooth component which was active when I filled that variable and after when I using that variable Bluetooth is off.
(Bluetooth deinint process)

and one another variable is also changed

it is bool type only and i am filling that variable two times and otherwise i am only comparing it.
(here is view how i am using it)

Address of this variables are

here is compiler configuration (i need to select optimize for size because i am getting iram overflow issue)

Sorry it was my mistake,illiglly accessing memory , was filling structure with -1 subsript, and that's why it was overwriting globle variables. solved it.


UIWebView deprecated issue in ObjectiveC Pod file

Actually, i'm using this pod file for my ObjectiveC projects. and, it's contains UIWebView classes.
As per the apple guidelines this is deprecated & you guys are fixed this issue on Swift pod file.
pod 'BFKit'
Could you please update this for ObjectiveC Pod file?

Done! You can use the new 3.0.0 version 😉

Thanks 😘
On Thu, Mar 19, 2020, 4:32 PM Fabrizio Brancati<EMAIL_ADDRESS>wrote:

Closed #41 https://github.com/FabrizioBrancati/BFKit/issues/41.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/FabrizioBrancati/BFKit/issues/41#event-3145059375,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKSYRTHGMINJZIV3XULGR2TRIH3UBANCNFSM4LPCR3KA
.


libsquoosh: Don't take the whole buffer from a view, respect byteOffset and byteLength

Describe the bug
libsquoosh accepts buffer views, and unwraps these:
https://github.com/GoogleChromeLabs/squoosh/blob/ad5002c79cccab7291b7bbb5d9f6270087ca6d4e/libsquoosh/src/index.js#L13-L15
Tools like webpack work with these kinds of buffers, where the typed array is a view on a larger buffer. However, the way libsquoosh converts views is problematic: it takes the whole underlying buffer.
To Reproduce
Create a larger buffer, load in a small PNG somewhere past the starting point, and feed that to the image pool; the image file I used (https://upload.wikimedia.org/wikipedia/commons/7/70/Example.png) is 2335 bytes small but embedded in an 8kb buffer:
const ImagePool = require('@squoosh/lib').ImagePool
const https = require('https')
const url = 'https://upload.wikimedia.org/wikipedia/commons/7/70/Example.png'

function httpsIntoBuffer(url, buffer, offset) {
  // put data from url into buffer starting at offset, return slice over
  // downloaded data
  return new Promise((resolve, reject) => {
    const req = https.get(url, (resp) => {
      pos = offset
      resp
        .on('data', (chunk) => { buffer.fill(chunk, pos); pos += chunk.length })
        .on('end', () => resolve(buffer.slice(offset, pos)))
        .on('error', (err) => reject(err))
      req.on('error', (err) => reject(err)).end()
    })
  })
}

async function mcve() {
  const pool = new ImagePool()
  try {
    // read PNG into section of larger buffer
    const buffer = Buffer.alloc(8192)
    const view = await httpsIntoBuffer(url, buffer, 1024)
    const image = pool.ingestImage(view)
    console.log(await image.decoded)
  } catch (err) {
    console.error(err)
  }
  pool.close()
}

mcve()

output:

Binary blob has an unsupported format

Expected behavior
I would expect the above to work, where libsquoosh uses:
 if (ArrayBuffer.isView(file)) {
   buffer = Buffer.from(file.buffer.slice(file.byteOffset, file.byteOffset + file.byteSize));

Passing in the explicit slice works when replacing the ingestImage() call with:
const image = pool.ingestImage(view.buffer.slice(view.byteOffset, view.byteOffset + view.byteLength))

which produces:
{
  bitmap: {
    data: Uint8ClampedArray(122464) [
      244, 209, 13, 255, 244, 209, 13, 255, 244, 209, 13, 255,
      244, 209, 13, 255, 244, 209, 13, 255, 244, 209, 13, 255,
      244, 209, 13, 255, 244, 209, 13, 255, 244, 209, 13, 255,
      244, 209, 13, 255, 244, 209, 13, 255, 244, 209, 13, 255,
      244, 209, 13, 255, 244, 209, 13, 255, 244, 209, 13, 255,
      244, 209, 13, 255, 244, 209, 13, 255, 244, 209, 13, 255,
      244, 209, 13, 255, 244, 209, 13, 255, 244, 209, 13, 255,
      244, 209, 13, 255, 244, 209, 13, 255, 244, 209, 13, 255,
      244, 209, 13, 255,
      ... 122364 more items
    ],
    width: 172,
    height: 178
  },
  size: 2335
}

Version:

OS w/ version: macOS 10.15
Node version: 15.14.0
npm version: 7.14.0

Unfortunately, due to a few people leaving the team, and staffing issues resulting from the current economic climate (ugh), the CLI and libsquoosh packages are no longer actively maintained. I know that sucks, but there simply isn't the time & people to work on this. If anyone from the community wants to fork it, you have my blessing. The squoosh.app web app will continue to be supported and improved.


Header revamp for the pagetree

New search/filter placement analog to django Filer
New sites/options handler
Added menu section "Main Navigation"
General spacing / shadow

Further tests for the pagetree follow in the next PR.

Coverage increased (+0.3%) to 88.717% when pulling d2e7a0a646aa3f604ccf3d92406c87e3a5af187c on FinalAngel:feature/pagetree-3.3 into 9c92ccb1aa713ddd6a7e4d0eaa43f5f82d7332a6 on divio:develop.

LGTM

anything left here?

@vxsx waiting on matthias response to @mkoistinen feedback. Or can we merge this @mkoistinen ?

ok, i'm merging this since i need this as a base for further things for the pagetree, I assume we will know when the feedback will be resolved and will act accordingly then


ocamlPackages.ca-certs: 0.1.2 → 0.1.3

Motivation for this change
Bugfix release: https://github.com/mirage/ca-certs/releases/tag/v0.1.3
Seems like I had bad timing with the initial PR…
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build ocamlPackages.ca-certs


Import default css is not working

Hi guys I'm trying to import  import 'react-input-range/lib/css/index.css' as you mention in readme.
Default style is not displayed at all. Is there something I can do about it or I should now please?
Thanks Mat

You might want to post an example repo to allow others to reproduce the issue. Otherwise, this is difficult to debug.

When using scss you can use @import "../../../node_modules/react-input-range/lib/css/index.css";

Had the same issue using webpack and css modules. If that is your setup, make sure you don't preprocess css in node_modules

Sorry guys, I was busy so much recently. I think it would just go now. I had different problem with me easy installing every npms to root directory instead of app/client so my local env was broken all the time. nothing new worked xD It works now :)

It doesn't seem to work using Next js by placing the import on _app.js


logos ISO has expired

The prerelease iso (https://mega.nz/#!CdUgDbBI!kNSGSsO0cCjD9lhWPDqtiw22rwsYr3hJsInPIoy-Hw4) has expired

This one should work ;)

On Sat, Feb 27, 2016 at 04:02:05PM -0800, Solal Nathan wrote:

This one should work ;)

Reply to this email directly or view it on GitHub:
https://github.com/install-logos/logos/issues/38#issuecomment-189751242
While you are at it, is there any way you can reset up the ricedb
website?

Well i'll try but i don't think i can. I don't have the ownership on anything. I could maybe host the same site/database on a server at home or a rented one but i don't think i would take that much engament on such a thing. If the owners could do something about it i would be nice but look like now one update it now sadly :(

it seems like something like sr.ht should work.
https://gogs.sr.ht/SirCmpwn/sr.ht

I have an old server laying around (running Debian stable), i'll try to do this when i have a bit of spare time. Thanks for the link ;)


TMDB provider is down again??

Description
No sources found for tv series and movies again on TMDB provider
Provider
TMDB
Reproduction steps
click on provider then selecr movie or tv show
error "no extractors found"
Operating system
Nvidia Shield Android TV OS version 11.0
Logs
No response
Please confirm the following

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
[X] If related to a provider, I have checked the site and it works, but not the app.

the issue is caused by vidplay keys endpoint which returns 404
val encodeId = encodeId(id, service.getKeys(key))

Yeah it's down

the issue is caused by vidplay keys endpoint which returns 404
val encodeId = encodeId(id, service.getKeys(key))

Is there a way to fix it?

How to fix?

For now, I recommend using the SFlix provider, instead of the TMDb provider.

Fixed for now on Release 1.6.7
Thanks to @rickhomes ❤ !

Fixed for now on Release 1.6.7 Thanks to @rickhomes ❤ !

This is great you got it working again. But this shows the importance of having multiple providers added to the TMDb provider, lemme know if you need any help finding more. Something that you can look into is sudoflix, it's open source and has actively maintained providers

It's down again. Getting http 403 error.

Might be the keys constantly changing. Maybe this is useful? https://github.com/Ciarands/vidsrc-keys

Might be the keys constantly changing. Maybe this is useful? https://github.com/Ciarands/vidsrc-keys

Thanks a lot, I couldn't fix it  before because of the keys so I'll see.
Right now it's because the website ask for a token, but I don't know yet how to find it.
Example: https://vidsrc.to/ajax/embed/episode/LoVcnrGv/sources?token=OTo39fhitRA%3D

Might be the keys constantly changing. Maybe this is useful? https://github.com/Ciarands/vidsrc-keys

Thanks a lot, I couldn't fix it before because of the keys so I'll see.
Right now it's because the website ask for a token, but I don't know yet how to find it. Example: https://vidsrc.to/ajax/embed/episode/LoVcnrGv/sources?token=OTo39fhitRA%3D

No problem.
Hmm yeh I'm not too sure about that, the link I shared they have a discord with devs in it, maybe they'd have an idea?

Will look into it

Fixed on Release 1.6.9
Thanks @oathzed for the link, it helped me A LOT

I just updated to 1.6.9 and still having the same issue, SFlix provider is returning error too

I just updated to 1.6.9 and still having the same issue, SFlix provider is returning error too

Fixed on TMDb provider as writed on the issue title
SFlix provider is this issue #148 not fixed yet!!!!

Still not working to the point its not usable. Very few movies/tvshows are working and not all episodes. One day they are playable another day they stop working

Still not working to the point its not usable. Very few movies/tvshows are working and not all episodes. One day they are playable another day they stop working

Works fine for me!! What movies not working and what is your error message

@stantanasi  Deadpool & Warefolf
Jessica johones season 2 episode 1 was not working and then next day it worked and then episode 2 is not working.


[5.3] require doesnt work with array validation anymore

Laravel Version: 5.3.10 / 5.3.11
PHP Version: 7.0.9
Database Driver & Version: NaN

Description:
required doesnt work on array upload.
Steps To Reproduce:
{{ Form::open(['route' => 'store.files', 'method' => 'POST', 'files'=>true]) }}
    {{ Form::file('image[]', ['multiple']) }}
    {{ Form::submit('submit') }}
{{ Form::close() }}

$this->validate($request, [
    'image.*' => 'required|image|max:2048'
]);

It's an array right? You want to validate that it's filled right? you can't go validate the members of the array because if the array is empty there won't be any members, instead you need to do:
'image' => 'required'

Which insures the array has members.
Please use the forums/stackoverflow, looks like most of your reporting issues are actually questions. Thank you!

actually it wasn't a question because this was working as usual in the past version v5.2,
also u could have simply add the functionality in the core by just checking is the array way empty or not and if yes then return the required msg :)


License?

What license is this under? (just randomly wandering through the repo list, not an issue right now)

https://github.com/spring/relayhost/blob/master/COPYING


Add utiity script for creating train.txt and valid.txt split files

The Darknet executable expects two files for specifying the training and validation datasets. To facilitate this specification file creation process we'll create a script that will simply list each file from separate training and validation directories that contain the images and write them into the appropriate files. We'll use train.txt and valid.txt as the default names but allow the user the option to specify the name/location of these output files. The CLI options should look something like this:
--train_dir: directory containing the training images (and corresponding Darknet files), required
--valid_dir: directory containing the validation images (and corresponding Darknet files), required
--train_file: path to the output training spec file, optional, will default to current working directory + train.txt
--valid_file: path to the output validation spec file, optional, will default to current working directory + valid.txt

Resolved by PR #10


feat: adding t-shirt sized typography (SDS-2513) #210

Covers:
SDS-2513 Adding t-shirt sized typography. Issue: #210
SDS-3259 Update Spectrum CSS Typography documentation: Spectrum CSS NextJS & legacy documentation
SDS-4405 Verify Spectrum CSS Typography Updates
SDS-4403 Split Typography documentation for Spectrum CSS
SDS-4460 Update Typekit / Adobe Fonts (Thanks @misterbrownlee )
SDS-4453 adobe-clean-ux support can be removed #416
Description

Updating Spectrum DNA to get t-shirt sized tokens.
Modifying processors.js to allow t-shirt sizes. Some minor restructuring of shared methods.
Adding t-shirt sized: sizes, margins and color
Enabling the debug statement the processors.js for colors and sizes.
Splitting Typography pages
Moving old Typography into Depreciated

For validation the following tokens can be searched:
Paperdoc summary of all changes: https://paper.dropbox.com/doc/Typography-Update--AopzJsKrFFzH_O8gexjcxdEdAg-bluAkl9HGke8WKkJvow2N
(updated t-shirt sized): https://git.corp.adobe.com/gist/nbaldwin/3364cf137d8bd50097c3bedbde9218e5
(original t-shirt sized): https://git.corp.adobe.com/Spectrum/spectrum-dna/pull/310
How and where has this been tested?

How this was tested:
Browser(s) and OS(s) this was tested with:

Ran Backstop.js tests a couple of times: Latest test is  https://spectrum-visual-regression.ci.corp.adobe.com/job/spectrum-css-vr-test-asset/157/artifact/spectrum-css/backstop_data/html_report/index.html
Tested on MacOS 10.14.5 (18F203)
on Chrome Version 80.0.3968.0 (Official Build) canary (64-bit)

Run:

yarn install
gulp dev

Visit locally:

New pages:
Body
Code
Detail
Heading
Old pages:
Typography page
Typography International page
Screenshots

To-do list

[x] If my change impacts other components, I have tested to make sure they don't break.
[x] If my change impacts documentation, I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.

[x] This pull request is ready to merge.

@bernhard-adobe please make sure that all font weights are included in the Typekit for Spectrum-CSS. It appears that weights heavier than bold are not included. This screenshot shows Heading XXL default (top) and Heading XXL Strong (bottom). By default, Heading is bold (font weight 700), but for <strong>, the font weight is black (font weight 900). DNA is providing the proper weight value, but it does not appear to be rendering.

Looks like this issue exists for the current typography as well.

Here is an additional reference comparing font weights as we have them defined in DNA and what is being rendered in Spectrum-CSS. I've added a hideous thick red border to the items that have incorrect font weights in CSS. Again, I believe this is due to the Typekit not pulling in all necessary styles/weights of Adobe Clean.

It also appears as though the internationalized fonts are not included either. When reviewing the Han examples, the letterforms "XXL" did not appear correct. Upon inspection I believe that this is not Adobe Clean Han. It would be good if we have all of the font families specified for CJK languages included in Spectrum-CSS's docs site so that we can accurately review the implementation.
Below is a screenshot of Spectrum-CSS Han example as-is:

And here is how it would appear when referencing Adobe Clean Han:

It appears none of the font stacks are coming directly from DNA. This becomes evident in the fonts listed to use for Arabic and Hebrew, which differ from what's defined in DNA. I would like to discuss this file: https://github.com/adobe/spectrum-css/blob/2f4fab674cc10f692a7ab3bcac799af5ad84a5fd/components/commons/fonts.css#L32 and how we can ensure that CSS pulls these fallback font stacks from DNA here: https://git.corp.adobe.com/Spectrum/spectrum-dna/blob/master/data/aliases/StaticAliases.mjs#L59

Please add examples of "Detail Light" to this section

Here is an additional reference comparing font weights as we have them defined in DNA and what is being rendered in Spectrum-CSS. I've added a hideous thick red border to the items that have incorrect font weights in CSS. Again, I believe this is due to the Typekit not pulling in all necessary styles/weights of Adobe Clean.

We will have to ask @misterbrownlee for either making the update or providing us the credentials.
Ticket here: https://jira.corp.adobe.com/browse/SDS-4460

It also appears as though the internationalized fonts are not included either. When reviewing the Han examples, the letterforms "XXL" did not appear correct. Upon inspection I believe that this is not Adobe Clean Han. It would be good if we have all of the font families specified for CJK languages included in Spectrum-CSS's docs site so that we can accurately review the implementation.
Below is a screenshot of Spectrum-CSS Han example as-is:

And here is how it would appear when referencing Adobe Clean Han:

This looks like another update to the Typekit instance.

Great and extensive work @bernhard-adobe and @NateBaldwinDesign.

@bernhard-adobe just to follow-up on the Slack conversation: for serif, we should use the classname spectrum-Typography--serif as a replacement for spectrum-Article.

blocked currently by the Typekit update: https://jira.corp.adobe.com/browse/SDS-4460

@bernhard-adobe now that the Typekit updated has been made, is there anything else remaining on this PR? If not, we can go ahead and merge and release it tomorrow!

@lazd  This is ready for review.
We should discuss if we should hide of of that depreciated typography as I feel that it might clutter up a bit the sidebar navigation.

@bernhard-adobe the sidebar clutter is real, but it shouldn't hold up this PR.
@GarthDB, if this is good to go in your opinion, I say merge it!

@lazd @GarthDB  I made the requested changes:
Examples are:
class="spectrum-Detail spectrum-Detail--XL"
class="spectrum-Heading spectrum-Heading-XXXL--light spectrum-Heading--serif"

I think it should be spectrum-Heading spectrum-Heading--XXXL spectrum-Heading--light spectrum-Heading--serif -- in SUIT, a single - denotes a sub-element, and modifiers can only be applied to the component classname.

@lazd I added those SUIT rules for  "regular, serif, heavy, light" as I believe those only modify the font-weight and style.
Mut I would propose to keep the other discrete rules to allow for an easier transition and be a bit more forgiving in case classnames are being missed or follow from an older standard. Let me know what you think.

@bernhard-adobe are you saying keep support for spectrum-Heading-XXXL--light in addition to spectrum-Heading--light spectrum-Heading--XXXL? If so, i must disagree, the former does not conform to the naming convention (it implies there is a subelement called XXXL and incorrectly places a modifier on the subelement).
Let's make this one clean and have it perfectly conform to our naming convention, that way we don't have to support anything else in the future.
Also, I'm on PTO so I'm away from my work computer, so can you clarify for me that this PR introduces T-shirt sizes for Typography in addition to the existing (twice deprecated) methods? If so, that's good -- we can do another 2.x release with all of the different Typography APIs together, and we can deprecate the old ones once and for all in the new year with a 3.x release.

I think we should consider supporting the spectrum-Heading-XXXL--light for legacy reasons. When converting over to the new Typography updates there will be plenty of cases were our consumers where expecting a single class name only (as it used to be) and might forget adding all the other heading, heading-light and serif classes. It would make using the new typography updates easier. what do you think @GarthDB ?

@lazd  Maybe you should consider reviewing this PR after your PTO? I don't think releasing any of this while you are on PTO is necessary, especially as there is no product need for the Typography updates during this week or shutdown.

@lazd  "can you clarify for me that this PR introduces T-shirt sizes for Typography in addition to the existing (twice deprecated) methods?" Yes. that's why we are keeping the numeric t-shirt sizes and the 2 other legacies. They are not removed from the sidebar.
I believe there is room for debate how far we are going to force the naming conventions and what that would mean for everyone moving to the new Typography V4.
The intend by me and @NateBaldwinDesign was to only touch the CSS classnames (aka implement the new Typography t-shirt size updates) but not having our user rewrite their HTML structure in order to implement the t-shirt sizes.

@bernhard-adobe The fact that we're keeping the existing classes for Typography gives us room to properly redefine the API according to the design API and our naming convention. I don't think it makes sense to violate our naming convention to preserve some semblance to a previous violation of our naming convention. If folks are touching their HTML to use the new styles, they'll be in there anyway and can make the necessary changes.

@lazd Maybe you should consider reviewing this PR after your PTO? I don't think releasing any of this while you are on PTO is necessary, especially as there is no product need for the Typography updates during this week or shutdown.

Yeah nah we won't release it before shutdown, but I wanted to make sure you had the insight you needed to be unblocked, hence the drive-by PR reviews :)

Yeah nah we won't release it before shutdown, but I wanted to make sure you had the insight you needed to be unblocked, hence the drive-by PR reviews :)

Thank you, I really appreciate that. It is really not my intention to pull you back into this while on PTO.
So far this PR has been fully unblocked by @GarthDB and @NateBaldwinDesign. Currently I am addressing your feedback which turns out to be a rather large refactor with a lot of opinions and a stronger impact in what it will require to adopt our typography updates. We could chat more about this in January to get a feel if this is the right way to get Typography V4 out of the door.
I have however made your requested changes.

@bernhard-adobe thanks for working to address the feedback. You're right, now that we have a design API for Typography that specifically defines the options for each Typography component, it is a rather large refactor to get it in line, but its important to do so for consistency (and important to stay in line with our naming convention in the process). We can discuss it after shutdown if necessary, and I'll hold further review until then.

This PR influenced quite a few components. See the difference: https://spectrum-visual-regression.ci.corp.adobe.com/job/spectrum-css-vr-test-asset/140/artifact/spectrum-css/backstop_data/html_report/index.html

we did some more extensive visual testing. the final visual changes are in this Backstop.js test:
https://spectrum-visual-regression.ci.corp.adobe.com/job/spectrum-css-vr-test-asset/157/artifact/spectrum-css/backstop_data/html_report/index.html

Once the analysis in my comment is double-checked, and #511 is merged, we can merge this!

we maybe should think about announcing this in the appropriate channels when we do a release or maybe before if we feel that this is a breaking change.

@bernhard-adobe since we kept all of the old classes intact, this is not a breaking change. That said, it's a great idea to annouce that Typography has finally caught up.
Additionally, it's a great idea to make this a breaking change in the future and remove the 2 other deprecated Typography implementations :)

I am working on correcting the component statuses and links on this PR and will merge shortly.

@lazd branched of this branch and created PR #523 to address some more documentation issues and bugs.


Test latest GHC versions

Changes are made so that the following GHC versions are tested:

GHC 9.2.7
GHC 9.4.4
GHC 9.6.1

The base and template-haskell dependency version bounds are updated to support GHC 9.6.1.  A method was added to Quasi in template-haskell-<IP_ADDRESS>, so a CPP-guarded definition for this method is added to the instance in QuasiMock to get rid of the missing method warning.
Note that support for GHC 9.6.1 requires updated explainable-predicates dependency version bounds (cdsmith/explainable-predicates#19).

Thank you.

I see that the CI failed to configure due to the explainable-predicates constraints.  (I tested using allow-newer in cabal.project.local.)
If you have a chance to release new versions (or make Hackage revisions) of explainable-predicates and HMock sometime, I would appreciate it.  Thanks!


Timestamps are incorrect in electricity-usage.csv

https://datasets.mlpack.org/examples/electricity-usage.csv
E.g.:
11/25/2011 21:00:00,0.36,1,0,0
11/25/2011 22:00:00,0.39,1,0,0
11/25/2011 23:00:00,0.37,1,0,0
11/25/2011 00:00:00,0.3,1,0,0
11/26/2011 01:00:00,0.3,1,0,0
11/26/2011 02:00:00,0.26,1,0,0
11/26/2011 03:00:00,0.27,1,0,0
The date should roll over at 00:00, but it rolls over at 01:00.

I also suggest using ISO-8601 for dates. Then, this error could have also been revealed by sorting the file and observing that the new order differs from the original order.

Thanks!  I fixed the file on datasets.mlpack.org. :+1:


inspector: init at 0.2.0

Description of changes
Closes https://github.com/NixOS/nixpkgs/issues/347834

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Closing since #349656 was merged.


Run it in a more permanent home

Currently this runs in a cronjob on the rust-bots EC2 instance, but we're trying to retire it. We should migrate it either to a scheduled job on Fargate, or on a periodic GitHub Actions job. Running it weekly on GHA seems like the easiest path.

Running on GHA weekly now.


some modes dont have potions spawning

*Oh! I found vendor potions in SX. I'm not finding them on B, but I transferred some over... they do nothing.

fixed. was missing an entry in the excel generator.
haven't tested but will let others test


Ensure that PORT environment variable behaves as expected

Describe what problem your feature request solves
We have an environment variable PORT that defines the port number that the TD server listens on. If this is changed it is probable that the front end will not be able to access the backend
Describe the solution you'd like
Ensure that the PORT environment variable behaves as expected
Additional context

Taking this on so that I can increase my knowledge of the code


[#466] Internationalization support

User story
See issue #466
Definition of Done
Misc.

[x] Update CHANGELOG
[x] Update documentation

In demo app

[x] Extract any localizable
[x] Management if needed of texts (singular)
[x] Management if needed of texts (plural)
[x] Management if needed of dates
[x] Management if needed of numbers
[x] Management if needed of currencies

In SDK

[x] Extract any localizable
[x] Management if needed of texts (singular)
[x] Management if needed of texts (plural)
[x] Management if needed of dates
[x] Management if needed of numbers
[x] Management if needed of currencies

Technical notes
Items / sections to check

[x] Colours
[x] Spacings
[x] Typography
[x] Orange Theme
[x] Innovation Cup Theme
[x] Banners Demo
[x] Bars - navigation
[x] Bars - tab
[x] Bars - tools
[x] Buttons
[x] Cards
[x] Chips
[x] Lists
[x] Progress indicators
[x] Sheets: Bottom
[x] Slides
[x] Text field
[x] Module - about
[x] Module - card colelctions
[x] About - design guidelines
[x] About - app news
[x] About - change log
[x] About - privacy policy
[x] About - temrs of services
[x] About - acessiblity statement
[x] Compare with Android version to have the same wordings
[x] Factorise the wordings to share keys

Tests

In a simulator or device with english as app language, run the app and walk through its features
Then change the language of the app to french, go back to the app: language must be applied, any wording must be displayed in french and no more in english

Tested on

iPhone 14 Pro (simulator) iOS 17.0

Note that topics related to RLT/LTR layouts, images, animations etc will be handled later.
This issue is only related to texts (dates, raw) and maybe currencies or figures if this responsibility is not in te customer side.

Pull request will be automerged because @ludovic35 is missing but approved offline the content of this evolution.


Munkireport: manual check reporting

We currently have a number of Macs for which the data shown in Munkireport is showing old versions. I am mainly concerned with Sophos anti-virus versions in this case.
We are therefore trying to resolve this by either ourselves or getting the users to connect to the network and run a Munki update cycle and hence a Munkireport cycle. The idea was this would refresh the data.
I was aware some reports might either only run on their own schedule e.g. possibly daily or if they detect a change but it seems many reports do not run on a manually triggered Munki update check. This includes of course Sophos, I can see the following messages in the log.
Jul 31 2019 10:42:12 +0100     Munkireport: Running sophos.py
Jul 31 2019 10:42:12 +0100     Munkireport: Manual check: skipping
This is currently on a Mac running macOS 10.14.6 with Munki <IP_ADDRESS>76 and Munkireport 4.0.38 listed for the client and the server showing <IP_ADDRESS>85
Now all the above may indeed be correct behaviour but it is making it harder to ensure all the Macs are running up-to-date Sophos software. Is there a way to configure which reports run also during a manual check? Should this one be triggered instead on detecting a change?

Moved to Google discussion group.


Use 10 minute average instead of instantaneous data

Someone requested we use the 10 minute average instead of the latest value. I think that'd be good, but I believe we'll need to poll ThingSpeak instead of PA for that. Unsure how hard that'd be to implement, but I wanted to make note of it.

Is there a -reason- they want the 10 minute averages?
One -possible- alternative to using the thingspeak API would be to compute our own rolling 5 or 10 minute average if we have the data and the sensor we're polling hasn't changed.

(I don't believe that there's no reason to want 10 minute averages or that those averages aren't better. I am absolutely not skilled in that side of things. But I'm a big fan of documenting the "why" of changes.

They said they preferred the 10 minute average to avoid sudden spikes/dips and smooth those out. I think that's fair, but also not a huge problem. I promised I'd document this but I'm not sure it's a huge priority.


versioning and bundling

Made version more bndtools build friendly, added maven-bundle-plugin and turned this into a bnd-built jar artifact.

Will send again.


Numbers returned by MarkLogic do not use scientific notation

Version of MarkLogic Java Client API
5.5.3
Version of MarkLogic Server
10.0-9.5
Java version
java version "<IP_ADDRESS>" 2022-08-18 LTS
Java(TM) SE Runtime Environment 18.9 (build <IP_ADDRESS>+1-LTS-1)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build <IP_ADDRESS>+1-LTS-1, mixed mode)
OS and version
OS name: Microsoft Windows 11
OS version: 10.0.22000
Input: Some code to illustrate the problem, preferably in a state that can be independently reproduced on our end
Write the following RDF data into MarkLogic:
<http://John> <http://have> 2.1123519E7 .

Then execute the following query:
SELECT *
WHERE { ?s ?p ?o. }

Corresponding Java code:
import java.io.File;
import java.io.IOException;
import com.marklogic.client.DatabaseClient;
import com.marklogic.client.DatabaseClientFactory;
import com.marklogic.client.io.*;
import com.marklogic.client.semantics.*;

public class MarkLogic_Test {
    static DatabaseClient client = DatabaseClientFactory.newClient(
            "localhost", 8000, "Documents",
            new DatabaseClientFactory.DigestAuthContext(
                    "root", "123"));
    static private GraphManager graphMgr = client.newGraphManager();
    static private String graphURI = GraphManager.DEFAULT_GRAPH;
    static private String tripleFilename = "./example.ttl";

    // Load managed triples from a file into a graph in MarkLogic
    public static void loadGraph(String filename, String graphURI, String format) {
        FileHandle tripleHandle = new FileHandle(new File(filename)).withMimetype(format);
        graphMgr.write(graphURI, tripleHandle);
    }

    // Evaluate a SPARQL query.
    public static void sparqlQuery() throws IOException {
        SPARQLQueryManager qm = client.newSPARQLQueryManager();
        SPARQLQueryDefinition query = qm.newQueryDefinition(
                "SELECT *\n" +
                        "WHERE { ?s ?p ?o . }"
        );
        JacksonHandle results = new JacksonHandle();
        results.setMimetype(SPARQLMimeTypes.SPARQL_JSON);
        results = qm.executeSelect(query, results);
        results.write(System.out);
    }

    public static void main(String[] args) throws IOException {
        loadGraph(tripleFilename, graphURI, RDFMimeTypes.TURTLE);
        sparqlQuery();
        client.release();
    }
}

Triple file:
example.ttl
<http://John> <http://have> 2.1123519E7 .

Actual output: What did you observe? What errors did you see? Can you attach the logs? (Java logs, MarkLogic logs)
The query result is:

?s
?p
?o

<http://John>
<http://have>
"21123519"^^xsd:double

Expected output: What specifically did you expect to happen?
The expected query result is:

?s
?p
?o

<http://John>
<http://have>
"2.1123519E7"^^xsd:double

Alternatives: What else have you tried, actual/expected?
The written data is expressed in scientific notation, but the data returned in query result is not in scientific notation. Will there be some inconsistencies?
Executing the SparQL query with Java Client API will get the unexpected query result. But executing it in Query Console will get the expected query result.
This query can also return the expected query result in Apache Jena and RDF4j.
Can someone give me an answer or a hint about it?

As noted in the stackoverflow post - https://stackoverflow.com/questions/75412052/numbers-returned-by-marklogic-do-not-use-scientific-notation - you're seeing behavior from the server based on the response type, where a response type of JSON results in a value that does not use scientific notation. The Java Client does not have any bearing on this.
I'm verifying with some colleagues that this is expected behavior and that the two different ways of representing the same value are not an issue. We'll respond to the stackoverflow post to confirm, as this does not appear to have anything to do with the Java Client.


Bulk Streaming

Hi !
Is it possible to stream a  bulk into elasticsearch?

Do you mind explaining your use-case a bit more?
By 'stream' bulk do you mean use a Stream of json bytes as the body for the bulk request? Or observing a stream IBulkOperations and sending those asynchronously?

observing a stream of objects  and send tthem in bulk
var size = 1000;
var pages = 100;
var seenPages = 0;
var numberOfDocuments = size * pages;
var documents = this.CreateLazyStreamOfDocuments(numberOfDocuments);
We can then set up our BulkAll() call as follows:

var observableBulkAll = client.BulkAll(documents, f => f
    .MaxDegreeOfParallelism(8)
    .BackOff(TimeSpan.FromSeconds(10))
    .NumberOfBackOffRetries(2)
    .Size(size)
    .RefreshOnCompleted()
    .Index(IndexName)
);
and then register an observer

var bulkAllObserver = new PartitionedBulkObserver(
    onError: (e) => { throw e; },
    onCompleted: () => doSomething(),
    onNext: (b) => Interlocked.Increment(ref seenPages)
);
observableBulkAll.Subscribe(bulkAllObserver);

Get access to write stream
this is not possible in the low level API although you can get direct access to the response stream
The lowlevel API only allows you to send byte[]
https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/post-data.html


feat: incorporate feedback from Macjonathan and Micha

Summary by Sourcery
Incorporate feedback by standardizing capitalization in acronym definitions and correcting grammatical errors in the documentation to improve clarity and consistency.
Enhancements:

Standardize capitalization in acronym definitions for consistency.

Documentation:

Correct minor grammatical errors and improve clarity in the documentation text.

looks good to me! nice that now there is a command for the tiny subset :D


Inquiry Regarding Paper and Code Adjustments for nuScenes Dataset

Hi,
First, I would like to express my appreciation for your impressive work. I have been exploring your paper and the associated code and have a few questions that I hope you could help clarify.
In the paper, you mentioned that setting N=3000 iterations strikes a balance between runtime and the quality of the attribution maps. However, I noticed that the demo is set at 6000 iterations. Could you help me understand the reason for this difference?
Additionally, I am keen on adapting your approach for the nuScenes dataset. As a preliminary step, I'm considering adjusting the parameters a, b, c, and lambda in the OccAM model configuration. I've seen that there's already a discussion on this topic under the thread "Questions about mean voxel density #3". In your response, you mentioned using a few hundred samples to determine the mean voxel density. Could you share more about how this was achieved? Did you utilize any open-source libraries or was it an entirely custom development?
Besides modifying the values of a, b, c, and lambda, are there other adjustments that are essential for successfully applying this method to the nuScenes dataset?
Thank you for your time and assistance. I look forward to your insights.
Best regard,
Kexuan Xia

Hi,
thank you for your interest in our work!
In the demo code, we used N=6000 to make the importance of each point even more prominent. However, the same gain in information about the important points can be obtained with significantly fewer iterations.
To determine the mean voxel density for a new dataset, the following steps should be performed on several hundred LiDAR frames (should be efficient for convergence of the fitted function):

Voxelization of the point cloud using spconv.
Calculation of the center coordinates for the non-empty voxels.
Computation of the pairwise distances between the center coordinates of all non-empty voxels using scipy.spatial.distance.cdist.
For all non-empty voxels, computation of the number of neighboring non-empty voxels within a given radius and normalization using the maximum number of possible neighbor voxels.
According to the distance between the LiDAR sensor and the corresponding non-empty voxel, this data is stored.

Finally, the distance range to cover is divided into 0.5m steps and the mean voxel density is calculated for each bin. These data points are then used to fit a function of the form f(distance) = 1 / (a * distance^2 + b * distance + c) using scipy.optimize.curve_fit.
The sampling probability, i.e. the probability that a voxel is not masked, is then computed as lambda * (1/f(distance)).
The specific value of lambda has been empirically determined to ensure that the average similarity score is about 0.3, as shown in Figure 8 of the paper.
In total, for a new dataset, the function with which the voxel density decreases with distance must be approximated (since it depends on the LiDAR sensor characteristics) and the parameter lambda (depending on the detector) must be set such that the detections begin to degrade due to the masking.
I hope this helps.
With best regards
David

Hi,
I would like to sincerely thank you for your detailed and patient answer!
I believe it's really meaningful for me to reproduce the determination of the mean voxel density step by step on KITTI. If the result keeps the same as yours, I am going to do it again on a new dataset. Unfortunately, I am stuck in the first step.
I take as input a point cloud "000001.bin" form KITTI, use the function PointToVoxel from spconv to voxelize it.
# read point cloud and drop the 4th column since intensity is not necessary for voxelization
source_file_path = 'demo_data/000001.bin'
if source_file_path.split('.')[-1] == 'bin':
    points = np.fromfile(source_file_path, dtype=np.float32)
    points = points.reshape(-1, 4)[:, :3]
elif source_file_path.split('.')[-1] == 'npy':
    points = np.load(source_file_path)[:, :3]
else:
    raise NotImplementedError

# number of total points
print("number of points: ", points.shape[0])

# transfer np into torch
device = torch.device("cuda:0") if torch.cuda.is_available() else torch.device("cpu")
points = torch.from_numpy(points).float().to(device)

# initialize PointToVoxel
point_to_voxel = PointToVoxel(
    vsize_xyz=[0.2, 0.2, 0.2],  # voxel size
    coors_range_xyz=[-80, -80, -10, 80, 80, 50],  # coordinate ranges
    num_point_features=3,  # number of point features
    max_num_voxels=20000,  # maximum voxels, the same as the setting of your work
    max_num_points_per_voxel=200,  # maximum points in each voxel, the same as the setting of your work
    device=torch.device("cuda:0")  # GPU
)

voxels, indices, num_points_per_voxel = point_to_voxel(points)

I have two questions about the output num_points_per_voxel:

To the best of my knowledge, the total number of the points should be the same or at least quite similar before and after voxelization. However, I saw that before voxelization it was 122k but only 71k after voxelization. If I set max_num_voxels=40000, the problem disappeared. But it should be max_num_voxels=20000 according to the config file kitti_pointpillar.yaml in your work.
Mean voxel density is defined as the mean percentage of non-empty voxels within 1 m, depending on the distance.So, if I understand correctly, there should be some voxels which contains no points. Which means there should be some zero-elements in the array num_points_per_voxel, but my output showed they were all non-zero in num_points_per_voxel. It really confuses me.

number of points:  122109
Number of Points after Voxelization: 71198
Number of zero voxels: 0

Thank you for reading this far. Wish you have a good day.
Best Regards,
Kexuan Xia

Hi,
Yes, that is correct, the number of points before and after voxelisation must be the same, i.e. the sum of num_points_per_voxel.
A few comments on this. First, it seems that you did not crop the point cloud according to the camera frustum as done usually in the pcdet dataloader. Second, I use the CPU implementation of spconv like pcdet, because I had some problems with the GPU implementation. And third, you can alternatively use the generate_voxel_with_id function (https://github.com/traveller59/spconv/blob/2.1.x/spconv/pytorch/utils.py) to check where the points are lost.
Regarding the second question: Yes, the majority of voxels should be empty. However, you can also use the number of all possible voxels within the radius to compute the density.
Best regards
David

Hi,
Thank you so much for your thorough and considerate response!
I did not crop the point cloud, I will do it to see if this is the only cause for the problem. Could you please tell me why 42 degrees is used to crop the point cloud? Is it because the same degrees were used for getting the images in KITTI?
I wish you have a wonderful week.
Best regards
Kexuan Xia

Hi,
the 42 degrees is just an approximation to make the demo work without the corresponding KITTI info files.
However, this is not a prerequisite for determining the voxel density, you just need to increase the number of maximum voxels in the voxel generator, as you already noted.
Best regards
David


ImportError: No module named 'stanfordnlp'

the issues i encountered are the module issue.
and i believe the owner should add the stanfordnlp install procedure in the tutorial

you can find the tutorial about stanfordnlp in  https://github.com/stanfordnlp/stanfordnlp.

pip install -i https://pypi.tuna.tsinghua.edu.cn/simple stanfordnlp

@zhayefei ERROR: No matching distribution found for stanfordnlp
i got setup install issue as same as your propose to run pip install stanfordnlp


Filter construction managers depending on auth

Apply filters depending on construction manager authentication. Eases implementation in the app.
For the construction_managers and construction_sites GET requests.
Marked as bug as the app depends on this behaviour for construction managers not able to associate themselves.

Resolved in master


⚠ Bofrost.at has degraded performance

In 6aa9d1b, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1088 ms

Resolved: Bofrost.at performance has improved in 0b7e3c0.


Update Graphite remote example for Graphite v1.1.x

This pull request updates the graphite remote write example to take advantage of Graphite's tagging support. It includes an extension to allow for remote-reads.
If tag support is not enabled there is no change in functionality.
If tag support is enabled, series will be written to graphite using the 1.1.x tagging support, and if a graphite web address is specified then remote read functionality will be enabled
https://graphite.readthedocs.io/en/latest/tags.html

Please let us know if there's anything we can do to improve the chances of this being merged.

The example remote adapter is to help users transition from the code that was previously inside Prometheus, and to serve as examples for how to write them. I'm not sure we're interested in actively tracking upstream features.

@brian-brazil to reduce confusion could we consider pointing users to more advanced alternatives if they need more features ?

Sure, the integration page on the website can point elsewhere.


is it possible to embed in-game python scripting in pyglet?

I was researching about this and saw a SO answer that says it might be possible, if it's already supported by pyglet, can you point to the doc that shows how? I couldn't find the doc on this after searching.

What do you mean by in-game scripting? You can check out exec and eval, but be very careful about how you use it.

why wouldn't it be possible? you'd likely have to create a Python plugin system though and an SDK for plugins to use.

yeah thanks guys, I've experimented with exec and eval. I think now I just needs to find ways to limit the in-game Python usage to use limited language functionality or else the game will just be a mess of security problem lol

tl;dr Read this article by Ned Batchelder.
To be honest, restricting the language is pretty hard. You'd need to somehow go against what exec and eval are supposed to do, that is, running a string as if it was part of the source code.
I haven't tried this, but you could run the code with a special set of variables (by passing a dict of names and their values). Something like exec(code, variables) where code is the string that you are gonna execute and variables is the dict.
Another restriction would be checking and stopping code that uses double underscores (__) because they allow you to access some implementation specific stuff that probably won't be documented and can change frequently.
There are probably a few ways to bypass them, such as calling a method in the standard library that is always imported and always raises an exception (codecs.Codec) and getting the builtins from the exception, or getting double underscores with getattr and a string like "\x5F\x5Fclass\x5F\x5F" (which is "__class__"). I haven't thought of a way to get past both, but some dude out there's gonna find a way.
Some useful links:
https://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html
https://www.reddit.com/r/Python/comments/hftnp/ask_rpython_recovering_cleared_globals/

you could sandbox something like micropython and build an interface if you want to give them access to certain things

I'll close this for now since it's more of a general Python thing, not specifically pyglet related.
Thanks everyone.


Option do disable the default context menu

When packaging a default webapp, then the context menu (at least on macOS) has 2 entries:

reload
back

I wonder if there is an option to disable the context menu by configuration?
Of course, I can implement that in JavaScript but I wonder if that is not like a common case, where people want to default to no context menu actions because their webapp does not have any own ones.

You can disable on linux with this: webview.connect_context_menu(|_, _, _, _| true);. See this page for more details on the linux implementation. Not sure how to do it on other platforms.

context menu can be disabled from javascript with:
document.addEventListener('contextmenu', event => event.preventDefault());
Until this becomes a critical feature that we really need it, I'll close this one

you can use this:
function disableMenu() {
if (window.location.hostname !== 'tauri.localhost') {
return
}
  document.addEventListener('contextmenu', e => {
    e.preventDefault();
    return false;
  }, { capture: true })

  document.addEventListener('selectstart', e => {
    e.preventDefault();
    return false;
  }, { capture: true })
}

disable contextmenu only release

you can use this:
`disableMenu() {
if (window.location.hostname !== 'tauri.localhost') {
return
}
document.addEventListener('contextmenu', e => {
e.preventDefault();
return false;
}, { capture: true })
document.addEventListener('selectstart', e => {
e.preventDefault();
return false;
}, { capture: true })
}`
disable contextmenu auto only release

you can use this:
disableMenu() {
if (window.location.hostname !== 'tauri.localhost') {
return
}
document.addEventListener('contextmenu', e => {
e.preventDefault();
return false;
}, { capture: true })
document.addEventListener('selectstart', e => {
e.preventDefault();
return false;
}, { capture: true })
}
disable contextmenu auto only release

Of course it's possible to do this in Javascript, but what if you only want to disable it for release build? There is no debug flag in Javascript, so you'd have to invoke another command to get debug flag from Rust first. This seems too much twisted logic for such a small feature.


Don't output received logs to stdout.

For my setup(forwarding journalctl -> sematext agent), this is causing an infinite loop of duplicate logs.

Please pull sematext/sematext-agent-docker:1.30.25 and provide feedback. So we could label it as "latest".

@megastef 1.30.25 appears to fix the problem on my end. Thanks!

Thank you for reporting this issue!