Sync ensemble forecasts on THREDDS

Description
ECCC produces ensemble forecasts and distributes them through GeoMET and their datamart. The original plan was to fetch data from GeoMET using WCS directly in netCDF format, but the metadata fields are empty and there is no scheduled fix for this.
The solution for now would be to fetch files from the "datamart".
Create cron job to do the following every day at 7h and 19h.

Copy to temporary directory files from

https://dd.weather.gc.ca/ensemble/geps/grib2/raw/??/???/CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5__P_allmbrs.grib2
https://dd.weather.gc.ca/ensemble/geps/grib2/raw/??/???/CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5__P_allmbrs.grib2

Convert files to netCDF(see below)
Copy netCDF files on THREDDS to Birdhouse/eccc/ensemble/geps
Clean old netCDF files

The ??/???? in the path refer to the 0-h of the forecast (00 or 12), and the forecast lead time (increments of 3-h then 6-h).
Also (Travis)

Create NcML to aggregate variables and time. Note that there are two time dimensions, one for the time the forecast was issued, and another for the forecast time.

If this all works fine, we would follow a similar model to host other ensemble products (REPS, GDPS, RDPS).
Notes
CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020110900_P000_allmbrs.grib2
_____________VAR______________________YYYYMMDDHH_PFTS_________
where YYYYMMDD is the date and time the forecast was made, and FTS is the forecast lead time. Now that for precipitation, the first forecast is at 003 because it is the cumulative precipitation from time 00.
Note that there are 65 3-h time steps followed by 25 6-h time steps.
Conversion to netCDF
$ pip install xarray cfgrib
import xarray as xr

def convert(fn):
  """Convert grib2 file to netCDF format.

  TODO: Add parameter to change output directory.
  """
  ds = xr.open_dataset(fn, engine="cfgrib", backend_kwargs={'filter_by_keys':{'dataType': 'cf'}})
  ds.to_netcdf(fn.replace(".grib2", ".nc"))

@huard python code for retrieving and converting should go in utilities?

I suggest to keep it separate from raven. I think it could be a single file, possibly in pavics-sdi.

ok

we may want to look at using a subscription service?  https://eccc-msc.github.io/open-data/msc-datamart/amqp_en/
seems a bit complex but could simplify things for other datasets longterm

For the subscription service :
a test.conf file with the following seems like it should work but I can't seem to get anything actually running/downloading
@tlvu @huard  anyone have time to take a second look at the documentation ?https://github.com/MetPX/sarracenia/blob/master/doc/subscriber.rst#revision-record
broker<EMAIL_ADDRESS>subtopic ensemble.geps.grib2.raw
directory /tmp
accept .*CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_.*_P.*_allmbrs.*.#
accept .*CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_.*_P.*_allmbrs.*.#

% sr_subscribe add test
% sr_subscribe start test

Is it possible that there is a hashtag missing after:
subtopic ensemble.geps.grib2.raw
subtopic ensemble.geps.grib2.raw#

In the examples they have it as so for the GDPS:
subtopic model_gem_global.25km.grib2.#
They also seem to recommend downloading multiple at a time:
instances 5
Their example file:
# Fichiers GRIB pour le SRPD 15km
# Pour plus d'information sur ce produit, veuillez consulter la documentation a
#  http://weather.gc.ca/grib/grib2_reg_15km_f.html

broker amqps://dd.weather.gc.ca/

# RDPS 15km GRIB files
# For more information about this product, please consult the documentation at
# http://weather.gc.ca/include/lang.php

#instances:
#    quantité de processus qui téléchargent simultanément.
instances 5
#instances: These files are large, so downloading them one after the other is slow.
# with higher number, download several at a time.

# la téléchargement de sarracenia est optimale pour beaucoup de petits fichiers. C´est
# miuex pour les gros fichier d´utiliser une téléchargeur binaire, tel que wget.
plugin accel_wget
# sarracenia built-in downloader works best for large numbers of small files.
# for larger files, a binary downloader will be faster.

#expire, en usage opérationnel, c´est mieux 1d (1 jour) pour celui-ci. Il faut que ça soit plus long que l´interruption le plus long qu´on veut toléré sans perte.
expire 10m
#expire, in operational use, use 1d (1 day) as it needs to be longer than the longest interruption in downloads we want to tolerate without dropping downloads.

subtopic model_gem_global.25km.grib2.#
accept .*

What ours would look like:
broker<EMAIL_ADDRESS>instances 5

plugin accel_wget

# Set to 10m for tests but they recommend longer, such as 1d.
expire 10m

subtopic ensemble.geps.grib2.raw#
directory /tmp
accept .*CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_.*_P.*_allmbrs.*.#
accept .*CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_.*_P.*_allmbrs.*.#

Hope this can be useful!
Thanks a lot for working on this!

@richardarsenault  did you actually get this to start downloading files?  Everything seems to start up fine for me (no errors) but am not seeing any actual files coming in ... will keep trying in a bit

No, I didn't try, I just looked at the docs.

Getting files now ... i think the accept lines should not use the end hashtag
broker amqps://dd.weather.gc.ca/

# GEPS Ensemble forecast GRIB files

#instances:
#    quantité de processus qui téléchargent simultanément.
instances 3
# instances: number of downloading processes. overlap multiple downloads for higher performance.

# le téléchargement de sarracenia est optimal pour beaucoup de petits fichiers. C´est
# mieux pour les gros fichiers d´utiliser un téléchargeur binaire, tel que wget.
plugin accel_wget
# larger files use binary downloader, rather than built-in processing.

#expire, en usage opérationnel, c´est mieux 1d (1 jour) pour celui-ci. Il
#        faut que cela soit plus long que l´interruption la plus longue tolérée sans perte.
expire 10m
# expire, in operations, needs to be longer than the longest expected interruption in service.
#       suggest 1d (1 day.)
directory /tmp/output
subtopic ensemble.geps.grib2.raw.#
accept .*TMP_TGL.*.grib2
accept .*APCP_SFC.*.grib2

Oh yeah, good catch, missed that. Good news then!
Please let me know what the format looks like (ex: does it all come in folders per forecast timestep? Is it devided by forecast issue time? etc.
Thanks! Good progress right here.

Pretty good progress here : with the subscription service running I should get new files pulled automatically (I will leave running over the weekend to make sure I get expected behavior)
For conversion : a cron job could be set up on the following python code (could be run multiple times a day -- performance is pretty good ... see notes below)
import xarray as xr
from pathlib import Path
import multiprocessing as mp
import itertools as it
import tempfile
import shutil
import time
jobs = dict(GEPS=dict(inpath=Path('/home/logan/github/raven/tmp/output'),outpath=Path('/home/logan/github/raven/tmp/output/netcdf')))
def main():
    for j in jobs:
        time1 = time.time()
        infiles = sorted(list(jobs[j]['inpath'].rglob('*.grib2')))

        # only convert files that do not exist already
        infiles = [i for i in infiles if not jobs[j]['outpath'].joinpath(i.name.replace('.grib2','.nc')).exists()]

        combs = list(it.product(*[infiles,[jobs[j]['outpath']]]))
        pool = mp.Pool(25)
        pool.map(convert,combs)
        pool.close()
        pool.join()

        print('done' , j , 'time : ', time.time()-time1)

def convert(fn):
  """Convert grib2 file to netCDF format.

  infile, outpath = fn

  if not outpath.exists():
      outpath.mkdir(parents=True)
  ds = xr.open_dataset(infile, engine="cfgrib", backend_kwargs={'filter_by_keys':{'dataType': 'cf'}})
  encoding = {var: dict(zlib=True) for var in ds.data_vars}
  tmpfile = tempfile.NamedTemporaryFile(suffix='.nc', delete=False)
  ds.to_netcdf(tmpfile.name, encoding=encoding)
  shutil.move(tmpfile.name,outpath.joinpath(infile.name.replace(".grib2", ".nc")).as_posix())
if __name__ == '__main__':
    main()

Notes:

Individual files are tiny :  running on my VM with 4 cores and ~8GB ram I could convert 25 concurrent files without any issues

With this setup total conversion of ~1200 files took ~4 minutes (with limited VM resources)

Adding zlib compression shrinks individual .nc to a fraction of .grib2 file size ( ~1/10th)

Question : how long of an archive do we want to provide?  Currently I see forecasts with dates of 20201110 to 20201112.  But files will continue to accumulate in the output netcdf folder unless I decide to delete older ones

@richardarsenault @huard  APCP files has data variable of paramId_0 is this as expected?
For reference TMP_TGL has t2m which seems more appropriate

Good question. I'll ask @julemai to chime in as she has more experience than I do using ECCC forecasts, I've always used versions that had been previously processed to netcdf so I am unfamiliar with the base product.
As for the number of days to keep in "reserve", I expect we won't need that many, maybe 2-3 tops. Eventually we might want to do something to bridge the gap between, say, the 2-3 week lag of Caspar automatic retrievals and the current-day forecasts, but I need to think about this a bit.

@tlogan2000 did the query perform as expected during the weekend? If so, can we start thinking about the format that the data will be stored under? i.e. ncml with a fixed path and filename related to the forecast date? If we can determine the best way forward I can adapt my forecasting codes ASAP.

I got new files over the weekend! We can maybe schedule a meeting this week to discuss output format?

APCP files has data variable of paramId_0 is this as expected?
For reference TMP_TGL has t2m which seems more appropriate

I've checked with @julemai and it seems that it is normal, ECCC simply does not provide a descriptive variable name for the precip.
However, the precip is in meters "m" and is cumulative. @huard, we can leave it that way in cumulative because Raven can handle it, but in that case we need to add a flag in the template, which I think can be done pretty easily?

The flag is :Deaccumulate. Somewhere in the same block where you put :TimeShift and :LinearTransform

backend_kwargs={'filter_by_keys':{'dataType': 'cf'}}

@richardarsenault  @huard PVI changing to {filter_by_keys:{'dataType':'pf'}}  let's me access 20 runs instead of one.  (My reduced filesize comment is most likely no longer relevant!!)
Q : Is there any documentation on this dataset anywhere?  I'm not sure what i'm pulling with'cf' versus 'pf' etc it would be nice to know

https://apps.ecmwf.int/codes/grib/format/mars/type/
cf = control forecast
pf = perturbed forecast

Yes that is correct. For now, pf is preferred, but eventually we might want to upgrade to both, to get the 21 member ensemble.

@huard Deaccumulate

@huard @richardarsenault   two ncmls to test / validate on the server:
https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/geps_forecast/2020112200_test.ncml
https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/geps_forecast/2020112212_test.ncml
I will be looking at cleaning up the code / deploying on the server with long next week sometime

I'm getting this error
>>> ds = xr.open_dataset("https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/geps_forecast/2020112212_test.ncml")
>>> pr = ds.pr.isel(lat=45, lon=450, member=0)
oc_open: server error retrieving url: code=? message="Error {
    code = 500;
    message = "NcSDArray InvalidRangeException=Illegal Range for dimension 0: last requested 96 > max 95";
}"

ds.pr.isel(lat=45, lon=450, member=0)

@huard This might be a thredds cache / bandwidth thing? Can you confirm that setting chunks='auto' solves issue?
ds = xr.open_dataset("https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/geps_forecast/2020112212_test.ncml", chunks='auto')

Same error.

Ok that's weird asi it seems to work on my end.. Will try to dig into it more
Another option : the NcML layer might be an unnecessary layer of complexity in this case.  There are only two files per forecast date after my conversion script is finished running ( 1 per variable) ...  Any objection to me merging tas, pr into a single .nc and  just accessing via thredds normally?

If it works on your end, the problem must be on my end. What is your version of xarray and pydap?

xarray == 0.16.1
pydap not installed?

I do get the error without applying chunking so there might be something wrong.

Getting your error today... something need to be tuned

Good, at least we get consistent errors.
ncdump -h says
dimensions:
	lat = 361 ;
	lon = 720 ;
	member = 20 ;
	time = 97 ;

So there seem to be a mistake in the time dimension interpreted by NcML.

I think I got it. The time steps for pr and tas might be different. pr is accumulated precip, so there is no first time step. There might be an additional time step at the end.

Yep. I think i might need to ensure that tas and pr files have identical / commeon timesteps before creating the ncml
timestep of individual files:
pr == 96
tas == 97

I suggest adding a pr timestep at the first step with all values set to 0.

sorry I'm late to the party, but yes, the precip has one fewer time steps as the accumulation is from timestep 0-3, so timestep 0 does not exist. Adding it manually would probably be the easiest and safest option in my opinion.

Also, not sure why but the times I get are really weird:
ds.pr.time

<xarray.DataArray 'time' (time: 97)>
array(['2020-11-22T12:01:04.000000000', '2020-11-22T15:00:16.000000000',
       '2020-11-22T17:59:28.000000000', '2020-11-22T21:00:48.000000000',
       '2020-11-23T00:00:00.000000000', '2020-11-23T02:59:12.000000000',
       '2020-11-23T06:00:32.000000000', '2020-11-23T08:59:44.000000000',
       '2020-11-23T11:58:56.000000000', '2020-11-23T15:00:16.000000000',
       '2020-11-23T17:59:28.000000000', '2020-11-23T21:00:48.000000000',...])

Is it possible to round them to the nearest hour? Clearly they're very close but this will be unusable I think for Raven, which needs exact timedeltas between timesteps.

Also, not sure why but the times I get are really weird:
ds.pr.time

<xarray.DataArray 'time' (time: 97)>
array(['2020-11-22T12:01:04.000000000', '2020-11-22T15:00:16.000000000',
       '2020-11-22T17:59:28.000000000', '2020-11-22T21:00:48.000000000',
       '2020-11-23T00:00:00.000000000', '2020-11-23T02:59:12.000000000',
       '2020-11-23T06:00:32.000000000', '2020-11-23T08:59:44.000000000',
       '2020-11-23T11:58:56.000000000', '2020-11-23T15:00:16.000000000',
       '2020-11-23T17:59:28.000000000', '2020-11-23T21:00:48.000000000',...])

Is it possible to round them to the nearest hour? Clearly they're very close but this will be unusable I think for Raven, which needs exact timedeltas between timesteps.

ok . I look at that while patching the precip timesteps

I suggest adding a pr timestep at the first step with all values set to 0.
@richardarsenault @huard
If first precip time step is all nan is that ok?

No, because the Raven model does pr(t+1) - pr(t) to find the precip flux.

No, because the Raven model does pr(t+1) - pr(t) to find the precip flux.

ignoring nans though?

I think I figured out how to pad with zeros anyway

New version to test..  I think everything is ok nb these are nto ncmls. I simply have combined tas, pr in a single netcdf.
A straight 1to1 ncml mapping could potentially be done if we want to bother... not sure if there is added value in doing that
https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/geps_forecast/CMC_geps-raw_latlon0p5x0p5_2020112200_allP_allmbrs.nc

https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/geps_forecast/CMC_geps-raw_latlon0p5x0p5_2020112212_allP_allmbrs.nc

Works like a charm on my end.
Agreed. No value in adding an NcML layer for now.
Thanks !

Works for me, I was able to access those data on THREDDS and write the forecasting code with success. So, good news overall!
Only thing is that, say for the 2020-11-23-00 forecast, the ncml only includes tas. Then, for Nov 24-26, there were no ncmls files.
I am guessing that this is just because the latest version did not have a chance to be rolled out and actually download and merge data since then?
Thanks!

@tlogan2000 Having second thought about NcMLs. The value of the NcML is in having a static link that does not change, but get's you the "latest" forecast. This is in contrast to the file names, that will change every day, no ?

@tlogan2000 It would be useful I think to have a "latest" link, either with NcML or straight netCDF symbolic link.

ok we can look into having a single 'current' ncml that changes underlying data location at each update.   The previous ncfiles can remain for ~3 weeks or so ... @richardarsenault said caspar update every 2 weeks or so ... so we shouldn't have any gaps

Works for me, I was able to access those data on THREDDS and write the forecasting code with success. So, good news overall!
Only thing is that, say for the 2020-11-23-00 forecast, the ncml only includes tas. Then, for Nov 24-26, there were no ncmls files.
I am guessing that this is just because the latest version did not have a chance to be rolled out and actually download and merge data since then?
Thanks!

@richardarsenault I stopped getting the updates occasionally as I was getting a lot of data .... I want to fine tune on just a couple of forecast dates then we will start 'production' where everything should keep coming in

2020-11-23-00 forecast tas only
I think you might have been accessing a 'running' conversion.... i might need to write to a tmp location then move to the thredds to avoid users seeing 'partially converted'  files like this

@richardarsenault @huard  couple of updates:

The subscription service seems a bit unreliable in the end... I ended up with a relatively large number of missing files in my download folder which were available on the datamart.   So  I have implemented a download function that can be called before conversion is done.  Disadvantage is that it does not run in the background but could potentially be called in a separate cron job every couple of hours if we wanted?

Thursday outputs are extended forecasts (twice as many time steps) ... do we keep all of them or do we want to limit to the 'normal' 96 timesteps

@tlogan2000 Alright, thanks for the update.

That's a bummer. Yes, perhaps a cron job that can do the job and runs regularly would be a simple enough option? I'll let @huard see what's the best approach.

I think we can keep the timesteps fixed and not use the extended S2 forecasts, I don't think it's that useful for now. Maybe eventually we'll want to add that option, but right now I would say it's out of scope.

Thanks!

@tlogan2000 I'll defer to Long on what should be the most robust way to fetch those files.

FYI I have meeting with Long tomorrow so will try to plan a transition to getting this running operationally

@huard @richardarsenault  Test version of 'latest' ncml here:
https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/testdata/geps_forecast/GEPS_latest.ncml

Works like a charm here.

Same here! works great!

Ping @cjauvin

@huard @cjauvin I think the final locations might change for the "current ncml" as well as the netcdfs.  I.e. Do we want this deployed on https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/catalog/datasets/catalog.html   with a new 'forecast' folder?
Right now I am dumping everything in 'testdata' in the parent birdhouse folder

@tlogan2000 How complicated would it be to also fetch the deterministic GDPS run from here: https://dd.weather.gc.ca/model_gem_global/15km/grib2/lat_lon/00/003/
for files (should be same file structure as GEPS):
CMC_glb_APCP_SFC_0_latlon.15x.15_2020120300_P003.grib2
CMC_glb_TMP_TGL_2_latlon.15x.15_2020120300_P003.grib2
And do the same ncml treatment as before?
This would allow me to also check the "deterministic forecast" box for the development phase.
Thanks!

This should be pretty similar to the geps so should be doable... I suggest getting the geps running smoothly on the server first then diving into this

perfect, thanks!


 TODO to follow up and re-enable them (would require a lot of "cleanup" in our scss)

Why is this breaking? It's not an API change that would cause issues w/ people trying to change from 0.9.3 to this update, just a dep version bump. Npm will handle it if you specify 0.9.4


chore(deps): update module github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper to v1.3.7

This PR contains the following updates:

Package
Type
Update
Change

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
patch
v1.3.5 -> v1.3.7

Release Notes

terraform-ibm-modules/ibmcloud-terratest-wrapper
v1.3.7
Compare Source
Bug Fixes

only output diff on update (#315) (43f68e5)

v1.3.6
Compare Source
Bug Fixes

deps: update module github.com/gruntwork-io/terratest to v0.41.8 (#314) (7c9354f)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 2.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


add a helm template to generate the label section

issue: https://github.com/rancher/rancher/issues/31667

Was this closed in favor or https://github.com/rancher/rancher/pull/31669 ?
@jiaqiluo

@nickgerace  Yes.
sorry for the confusion.


Remove empty tags when popover title is not defined

The problem
When popovers are defined without title, the h3 tag is still there as empty.
Fixes #29127
The solution
The solution I found was simply to display another template when title is not defined.
Feel free to comment if you have any suggestions :)

I think we shouldn't add the title in this case, not add it and then remove it.

I think we shouldn't add the title in this case, not add it and then remove it.

Yes that was what I wanted to do initially but the solution had a big impact. But I am going to modify my code and use this solution which is better. Thanks!

Hi @charlottedestrait,
Thanks for the PR
Can you add unit tests please?

Thanks for the remark, I added them

Is this too stale to bring back? Any thoughts @twbs/js-review on how to proceed?

Is this too stale to bring back? Any thoughts @twbs/js-review on how to proceed?

@mdo I think we need a simpler solution.

@mdo I think we need a simpler solution.


Launch and tasks run correctly but all panes are empty

OS: macOS Monterey 12.4
VSCode version:  1.69.1
CodeLLDB version: 1.7.0
Compiler: nim 1.6.6
Example code:
import rdstdin, strutils

let str = readLineFromStdin "Input a string: "
let num = parseInt(readLineFromStdin "Input an integer: ")

launch.json:
{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "lldb",
            "request": "launch",
            "name": "nim: debug current file",
            "preLaunchTask": "nim: build current file (for debugging)",
            "program": "${workspaceFolder}/bin/${fileBasenameNoExtension}",
            "args": [],
            "cwd": "${workspaceFolder}",
        }
    ]
}

tasks.json:
{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "nim: build current file (for debugging)",
            "command": "nim",
            "args": [
                "compile",
                "-g",
                "--debugger:native",
                "-o:${workspaceRoot}/bin/${fileBasenameNoExtension}",
                "${relativeFile}"
            ],
            "options": {
                "cwd": "${workspaceRoot}"
            },
            "type": "shell",
        }
    ]
}

See verbose log:
codelldb.log
Code compiles and runs normally, but all panes are empty:

Why do you think this has something to do with CodeLLDB?   Are you able to debug your binary with CLI lldb or gdb?

@vadimcn Sorry :(
This is actually an issue I have with LLDB.


bucket should be optional for s3

Currently if you don't pass the bucket it fails with error.
  let mut builder = opendal::services::S3::default();

  builder
      .endpoint("https://....")
      .root("/")
      .access_key_id("..")
      .secret_access_key("..");

  let op = Operator::new(builder).unwrap().finish();
  let result = op.stat("/").await.unwrap();
  dbg!(&result);

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ConfigInvalid (permanent) at  => The bucket is misconfigured

Context:
    service: s3
', src/main.rs:19:37

I think bucket should be optional. This means calling / would list buckets and /bucket-name would list of items inside the bucket and so on.
If it is not optional I need to have a logic on my side to parse the path and set the bucket which tends to be a bit more work on my side.
Similar to builder.allow_anonymous() would be good to have builder.allow_bucket_in_path() method to opt in.

I think bucket should be optional. This means calling / would list buckets and /bucket-name would list of items inside the bucket and so on.

Thanks for the suggestion. But OpenDAL can't support operations like list_bukcets. It's required to have bucket set in builder.


[SPARK-16995][SQL] TreeNodeException when flat mapping RelationalGroupedDataset created from DataFrame containing a column created with lit/expr

What changes were proposed in this pull request?
A TreeNodeException is thrown when executing the following minimal example in Spark 2.0.
import spark.implicits._
case class test (x: Int, q: Int)

val d = Seq(1).toDF("x")
d.withColumn("q", lit(0)).as[test].groupByKey(_.x).flatMapGroups{case (x, iter) => List[Int]()}.show
d.withColumn("q", expr("0")).as[test].groupByKey(_.x).flatMapGroups{case (x, iter) => List[Int]()}.show

The problem is at FoldablePropagation. The rule will do transformExpressions on LogicalPlan. The query above contains a MapGroups which has a parameter dataAttributes:Seq[Attribute]. One attributes in dataAttributes will be transformed to an Alias(literal(0), _) in FoldablePropagation. Alias is not an Attribute and causes the error.
We can't easily detect such type inconsistency during transforming expressions. A direct approach to this problem is to skip doing FoldablePropagation on object operators as they should not contain such expressions.
How was this patch tested?
Jenkins tests.

Test build #63786 has started for PR 14648 at commit 2e0d7d6.

Test build #63786 has finished for PR 14648 at commit 2e0d7d6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63786/
Test PASSed.

Merged build finished. Test PASSed.

cc @cloud-fan

Test build #63838 has started for PR 14648 at commit 4094b43.

Test build #63838 has finished for PR 14648 at commit 4094b43.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63838/
Test FAILed.

Merged build finished. Test FAILed.

retest this please.

Test build #63845 has started for PR 14648 at commit 4094b43.

Test build #63845 has finished for PR 14648 at commit 4094b43.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63845/
Test PASSed.

Test build #63892 has started for PR 14648 at commit 0008c3e.

Test build #63892 has finished for PR 14648 at commit 0008c3e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63892/
Test PASSed.

Merged build finished. Test PASSed.

Test build #63962 has started for PR 14648 at commit b688abf.

Test build #63962 has finished for PR 14648 at commit b688abf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63962/
Test PASSed.

Merged build finished. Test PASSed.

thanks, merging to master and 2.0!

Thanks for review.


Crash while parsing MIDI file

I have a lot of MIDI files working. But one of them gets a crash.

Originally posted by @SinanKarasu in https://github.com/orchetect/MIDIKit/discussions/171#discussioncomment-5286192

Fixed in 0.8.8


ED25519_public_from_private() sometimes returns wrong result

As noted in #11600 there seem to be cases when ED25519_public_from_private() produces the wrong result for the SHA512 hash of the private key.
I've done some local testing involving using both the "old" and "new" codepaths to hash into separate output arrays to confirm that the "new" path is producing an incorrect result.

As noted in #11600, I think this is actually a problem in TLSProxy and not a problem with Ed25519 itself at all.

Closing, as this does not look like a library issue anymore.


Companion command to osf clone

Related to #18: it would be useful to have a simple command that is the opposite of osf clone. I thought about integrating it into osf upload -r but I think the semantics are sufficiently different (for example you don't really want to specify a destination storage, it should use the same as a file currently uses).
Also need to think about how to handle the case when a local file is different from what is on OSF. There are versions on OSF so we might be able to figure out something smarter than just "remote file is different, overwrite?".
Maybe this is the time to switch to osf clone to make the initial copy, osf fetch to update local from OSF and osf push to update OSF from local. Following the git names/pattern a bit.

Maybe this is the time to switch to osf clone to make the initial copy, osf fetch to update local from OSF and osf push to update OSF from local. Following the git names/pattern a bit.

+1
would 'osf clone' create a .osfcli.config file for you?

would 'osf clone' create a .osfcli.config file for you?

:+1:


Add support for Danmakufu

Was going to contribute a yin-yang icon SVG as this is what the filetype's icon would be, but seeing as one already exists for YANG I guess that isn't really necessary.

Agreed. FYI, contributions after a new release go under a new section in the changelog called "Unpublished", but I won't chide you for not knowing that.

Ah, ok. Duly noted. Thanks!

@drakeirving I noticed this while perusing the recently-added packages:

The file-icons Atom package has icon support for .dnh files, so if you are interested, check it out.

I should point out that changes to the codebase aren't reflected immediately in downloads. When enough changes have been made, the maintainer "cuts" a release, sending a snapshot of it to be distributed (which we all know as versions).
So technically speaking, the package has no support for .dnh files yet. It will in the next release. Until then, anything that hasn't been featured in a release is listed under the Unpublished section of the CHANGELOG file.

I'm aware; I had made it clear within the circles that would use this that the changes were not in a release version yet, and they would have to clone the repo themselves if they wanted the functionality right away. Thanks for the concern!


Use zookeeper 3.4.12

Zookeeper version 3.4.11 is removed in apache repositories:
http://apache.mirrors.tds.net/zookeeper/
zookeeper-cookbook should use 3.4.12 instead.

Released as v11.0.0

@Seitanas and thanks for opening this issue!


add some packages to skip codecov check

xref #8568
Add some packages to codecov.skip to skip codecov check:

In istio.io/istio/security/pkg/nodeagent/model packages, just a struct :https://github.com/istio/istio/blob/master/security/pkg/nodeagent/model/secret.go
,so it's no need to match the codecov requirement.

In packages, just two mashall function, no need also:
https://github.com/istio/istio/blob/master/istioctl/pkg/util/clusters/wrapper.go

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: costinm
If they are not already assigned, you can assign the PR to them by writing /assign @costinm in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Hi @stewart-yu. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

I don't understand why these should be excluded.
The struct should not require any code coverage, so why exclude that?
And why shouldn't those marshaling routines be tested?

In deed, the struct should not require any code coverage, but the packages need. From the codecov CI, some packages are not match the covrequirement defined in https://github.com/istio/istio/blob/master/codecov.requirement
W0915 13:03:41.186718   86416 main.go:296] Failed to upload coverage, missing build info
E0915 13:03:41.186738   86416 main.go:303] Following package(s) failed to meet requirements:
	Package Name		Actual Coverage		Requirement
E0915 13:03:41.186751   86416 main.go:305] istio.io/istio/security/pkg/nodeagent/caclient	57.70	80.00(default)
E0915 13:03:41.186764   86416 main.go:305] istio.io/istio/pilot/pkg/serviceregistry/kube	63.30	64.00
E0915 13:03:41.186775   86416 main.go:305] istio.io/istio/security/pkg/nodeagent/model	0.00	80.00(default)
E0915 13:03:41.186797   86416 main.go:305] istio.io/istio/istioctl/pkg/util/clusters	0.00	80.00(default)
E0915 13:03:41.186812   86416 main.go:305] istio.io/istio/security/pkg/nodeagent/cache	74.30	80.00(default)

In generate, thoes function will test in resource convert. I'll check it deeply later, if done, will ping you back. Thanks


Getting error "signature invalid" when trying to call my .net core api from this sample

Anyone know why I get the error "signature invalid" when trying to call my .net core api from this sample? I'm passing the access token in the header of the fetch() like other documents have suggested. I created a Azure app registration for my API and exposed the api etc. However, something is wrong with token or how I'm trying to validate it on my .net core api. The api works fine if I disable the [authorize] attribute on my class. Any suggestions would be greatly appreciated as I am suck.
.net core api startup.cs snippet:
services.AddAuthentication(AzureADDefaults.BearerAuthenticationScheme)
.AddAzureADBearer(options => Configuration.Bind("AzureAd", options));
react spa snippet:
// Get the user's access token
var accessToken = await this.props.getAccessToken(config.scopes);
    const startDateEncoded = encodeURIComponent('2018-02-01');
    const endDateEncoded = encodeURIComponent('2018-02-07');
      debugger;
    const response = await fetch(`https://localhost:44307/api/Amounts?StartDate=${startDateEncoded}&endDate=${endDateEncoded}`,{
        headers: !accessToken ? {} : { 'Authorization': `Bearer ${accessToken}`}
    }
    );

Hi @jpro1001 this is beyond the scope of this sample. I'm going to close this as an issue. Off the top of my head, you can't use an access token for Graph (the aud parameter inside the token is set to https://graph.microsoft.com) to access any other API, including your own. You should be getting an access token for scopes defined by your API. The whole process is outlined in https://docs.microsoft.com/azure/api-management/api-management-howto-protect-backend-with-aad.


Number of dependencies

This issue is part of this JOSS review
The package has many dependencies. In my experience, the more dependencies you have, the more issues about installation you will get. Therefore I'd suggest removing some dependencies if possible. E.g. Cairo is only needed for having a nice PNG, which can maybe be avoided? I could not find any call to purrr (admittedly I didn't search for a long time) except one that is commented.

I think the dependency on plumber is unnecessary. It's nice to mention its possible use, but you can have the code describing it in not evaluated chunks/doc and avoid the dependency.


plugin: panic: plugin was built with a different version of package

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go version go1.11 darwin/amd64
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN="/Users/juhwany/go_workspace/bin"
GOCACHE="/Users/juhwany/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/juhwany/go_workspace"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/juhwany/gos/go_server/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/mk/2tyywdsn0jq6l0vl8094tss80000gp/T/go-build380269681=/tmp/go-build -gno-record-gcc-switches -fno-common"
What did you do?

Clone the blow repo in each directory. (not under GOPATH)
server (main program) : https://github.com/juhwany/go_server
plugin : https://github.com/juhwany/go_plugin

On same machine, build server & plugin in each directory
Build server : go build -o main
Build plugin : go build -buildmode=plugin -o plugin.so

Copy plugin.so to server directory and run
./main

See error message when main program try to open plugin.

panic: plugin.Open("plugin"): plugin was built with a different version of package github.com/juhwany/go_server/v2/apis

What did you expect to see?
Main program should load plugin.so successfully.
I could not understand why it fails even though the referenced package version is same between main program and plugin.
Am I using plugin in wrong way?
What did you see instead?
plugin was built with a different version of package error message

Related (dup?) #26759 .

@randall77
No. I think this issue is slightly different from  #26759 . I didn't build main program and plugin under GOPATH.  And the the above sample projects use GO module for specifying dependency package. Go version is also different.

I do think they are all related though, including https://github.com/golang/go/issues/27062.  I think it all boils down to changing (absolute) paths at build time between the plugin build and the app build.

CC @ianlancetaylor @cherrymui

I'm not sure if anyone's already mentioned it, but putting the common(apis) package to a separate module outside the server one will work. It's definitely cumbersome, but can be used as a workaround until this issue is fixed.

The command "go list -m -json all" shows version of go_server on go_plugin, but it's not defined on go_server itself. I have a similar problem using multiples modules on the same repository using "replace" directive. I don't know how to have the same version of the module inside and outside the repository.

In #28983, @bcmills suggests to build plugins from within the main module.
// server/go.mod
module github.com/juhwany/go_server/v2
require github.com/juhwany/plugin v0.0.0
replace github.com/juhwany/plugin => ../go_plugin

$ go build -buildmode=plugin github.com/juhwany/plugin
$ go run main.go
DoSomething called!

Just throwing in my two cents here. Both the @tamasf97 and @oszika are workarounds.
However, @tamasf97's workaround only works if you can factor the shared code out without creating a circular dependency (not always possible).
@oszika's workaround using the "replace" directive is pretty inconvenient for clients of the shared code, as you need that local repo.

We are unable to run our plugin-based service with gomod due to this issue.

We are experiencing this issue after implementing go.mod in both the server and plugin

@itsmontoya, @dhalman: what specific commands did you run to build the plugin?
(Did you follow the approach in https://github.com/golang/go/issues/27751#issuecomment-443452815? Did you build with -trimpath? Both of those may be necessary.)

@bcmills we will try -trimpath and the other approach and get back to you ASAP! 👍

added -trimpath and the error moved from our internal lib to now:
Fatal error encountered: error encountered while initializing service: error loading plugins: erorr initializing plugins: plugin.Open("plugins/thePlugin"): plugin was built with a different version of package internal/cpu

@dhaimeen We've used the Go plugin API for quite a while now (over a year) and the error "different version of package internal/cpu" occurs when you've not used the -trimpath flag on both the shared object you build (plugin) and the main binary. Both bundle a copy of the Go runtime (although there's an open issue on the tracker about not packaging a copy of the Go runtime with plugin code). Hope this helps.

The main bin solved it for us in a manual build process. Thank you!!
We are now running into an issue where it appears -trimpath isnt working as expected when running from a command in the .Exec(...) env
The .so files produced by the exec command differ from the files produced by go build in the cli

Ok, I think we have determined the difference:
Successful:
cd <path-to-plugin>
go build -trimpath -buildmode=plugin -o <filename>

Unsuccessful:
go build -trimpath -buildmode=plugin -o <filename> <path-to-plugin>

Are there any ways to determine why the latter fails to produce the same output as the former?
Thanks!

@dhalman I think you can try using go build's -x flag, e.g.
go build -x -trimpath -buildmode=plugin -o <filename>

to print out how the go command invokes the compiler and the linker, and how it differs.
Maybe you want to do a go clean -cache first.

@dhalman, is the plugin in the same module as the main binary? If not, the module metadata could be what's throwing it off.
Do you have any cgo dependencies? That could also do it (#36072).

I have a collection of Go plugins I need to package and distribute. When building and running on my machine everything works as expected. However plugins don't work on other machines.
Here is the command I execute:
$ GO111MODULE=on go build -trimpath -buildmode=plugin -o ./my-plugin.so ./my-plugin-directory

I tried running the parent application calling the plugins both with and without -trimpath. Nothing works.
How can I fix this behavior?

@bcmills I got caught up on this one with modules - the problem can occur when you import a package from the same module in the "loader," and you import the same package but from a different module in the "plugin."

We have developed an open source solution for syncing go.mod repositories across large projects with internal dependencies. We find it especially useful for managing large  collections of plugins along with their private imports.
Happy to share it if requested, but would not want to advertise in this forum

@paralin Do you have a working and a non-working example so I can try to reproduce your solution? I am not sure to fully understand.

Any process?

I encountered the same issue while trying out this repo: https://github.com/vladimirvivien/gosh
The weird things happen only in VSCode debugging mode.
To reproduce it, this is the process:
go version      # go version go1.16 darwin/amd64
dlv version     # Version: 1.6.0

git clone https://github.com/vladimirvivien/gosh
cd gosh

go mod init github.com/vladimirvivien/gosh  # to fix the running issue

This is all good:
go build -buildmode=plugin -o plugins/splash_command.so plugins/00_splash.go
go build -buildmode=plugin -o plugins/sys_command.so plugins/syscmd.go
go run gosh.go  # all good

But when try debugging by setting the breakpoint at /gosh.go, line 59, I always get an error:
"plugin.Open("plugins/splash_command"): plugin was built with a different version of package runtime/internal/sys"

If I try this instead by adding the -trimpath:
go build -trimpath -buildmode=plugin -o plugins/splash_command.so plugins/00_splash.go
go build -trimpath -buildmode=plugin -o plugins/sys_command.so plugins/syscmd.go
go run gosh.go

I would get this error:
failed to open plugin splash_command.so: plugin.Open("plugins/splash_command"): plugin was built with a different version of package internal/unsafeheader
failed to open plugin sys_command.so: plugin.Open("plugins/sys_command"): plugin was built with a different version of package internal/unsafeheader

Why?

We have developed an open source solution for syncing go.mod repositories across large projects with internal dependencies. We find it especially useful for managing large collections of plugins along with their private imports.
Happy to share it if requested, but would not want to advertise in this forum

Hey @dhalman,
could you post the link to the open source syncing solution of yours. I find developing any a bit non-trivial plugin for any a bit non-trivial app a total nightmare. Maybe your solution might help me out. :)
Cheers

@kriptor i havent tried it in a while, but im here to help support if you do end up using it
https://github.com/gomuserver/gomu

Hello,
UP on this subjecT?

I posted some comments here about this issue previously, right now you can view the workarounds I've built as well as a work in progress hot reloading system (which is hampered by this bug) here: https://github.com/aperturerobotics/controllerbus/tree/master/hot

Thank you @paralin , but my problem is ... very ennoyinh.
Package A include C version 1.1 ( go mod used ).
Package B ( the plugin ) include C version 1.1 go mod used.
Result, Plugin B was buillt with a different version of C.
My "workaround" was to remove the go.mod used in the B package to make it works. Very dirty solution in my opinion.

plugin was built with a different version of package internal/unsafeheader
Currently running into this. I don't see how "plugin" can be useful if you can't build plugins in different Go modules from the loader.

In #28983, @bcmills suggests to build plugins from within the main module.
// server/go.mod
module github.com/juhwany/go_server/v2
require github.com/juhwany/plugin v0.0.0
replace github.com/juhwany/plugin => ../go_plugin

$ go build -buildmode=plugin github.com/juhwany/plugin
$ go run main.go
DoSomething called!

I had initially built the plugin with the same go.mod file as the main module. I encountered the issue moved plugin into its own module. I tried --trimpath to build both the plugin and main module but did no succeed. The reference mentions the directive.
@juhwany  Does this issue still exist for you?

Apparently disabling optimizations and inlining -gcflags="all=-N -l" "resolves" the issue for me, but the output files are huge :(
go build -buildmode=plugin -gcflags="all=-N -l" -o plugins/test.so plugins/test.go

I have basically tried everything and in the end I have this error: plugin was built with a different version of package internal/goarch". I am using exact same dependencies, go version, built environments, flags, etc...

Well, I am still stuck with the same problem. Not sure if there will be any update on this issue. Exactly for this issue we had to choose another language for the modular system we were working on.

I had to move Vroomy away from Plugins which was unfortunate.

Any update on this issue

Closing as I don't think there is any specific issue with the plugin package itself, it all appears to be application build environment differences.


Add description field to Module Resource and update local seed data.

Can be marked ready for review after final descriptions provided. See #126.

👌🏻


cron atty::is(atty::Stream::Stdin)

I have a program that reads from stdin only when  isatty returns false
let input_stdin = !atty::is(atty::Stream::Stdin);

When running this from a cron it always returns true like if there were something in stdin but there is nothing.
use atty::Stream;

fn main() {
    if atty::is(Stream::Stdout) {
        println!("I'm a terminal");
     } else {
        println!("I'm not");
     }

     let input_stdin = !atty::is(atty::Stream::Stdin); // isatty returns false if there's something in stdin.
     if input_stdin {
         println!("stdin"); // how to prevent this to happen when calling it from a cron
     }

}

Any idea how to prevent this so that I could call my app from a cron job and distinguish when there is something in stdin and not?

What cron-daemon are you using?

I am using the cron that comes by default in the latest FreeBSD: https://www.freebsd.org/cgi/man.cgi?cron(8)

First of all: isatty doesn't actually tell you whether your process does or can receive input via stdin but whether stdin is connected to a tty. There is no way that I'm aware of that a process can detect whether it may receive any input from stdin (if it is open).
Assuming you do want to know whether the process was spawned from a terminal and not by your cron: isn't your logic backwards?
let input_stdin = !atty::is(atty::Stream::Stdin); // isatty returns false if there's something in stdin.
if input_stdin {
    println!("stdin"); // how to prevent this to happen when calling it from a cron
}

atty::is returns true if stdin is connected to a tty, which should not be the case for a cronjob, so we'd expect input_stdin to be true when run as a cron-job. Or does your program actually expect input only when it is not run from a terminal?

It has been working "fine" (https://github.com/s3m/s3m/blob/master/src/s3m/options.rs#L211) but I never tested within a cron.

atty::is returns true if stdin is connected to a tty

I have been using it to detect if something is in the stdin so that I can read the input and stream it, but maybe is not the way I should be doing this.

I have been using it to detect if something is in the stdin so that I can read the input and stream it, but maybe is not the way I should be doing this.

Yes, that's indeed not how you do this. Tools typically just read until the "end of file" in such situations.

It's not quite that simple. It's pretty typical for tools to change their behavior based on whether they believe stdin is readable or not. Whether stdin is attached to a tty or not is just one part of that calculation.
Here's the logic that ripgrep uses as an example: https://docs.rs/grep-cli/0.1.6/src/grep_cli/lib.rs.html#190-213
Logically speaking: read from stdin if it's believed to be readable, where "believed to be readable" means "it's not a tty and has a file type amenable for reading."

I will give it a try, thanks for sharing 👍

Within a cron, I still get that there is something from stdin,   wondering if is reliable to do something like:
let mut line = String::new();
std::io::stdin().read_line(&mut line).unwrap();
if line.len() > 0 {
// something to read from stdin
}

What could be the cons?

From a cron, I still getting like if there is something to read from stdin,
wondering what could be the cons of using something like this:
let mut line = String::new();
std::io::stdin().read_line(&mut line).unwrap();
    If line.len() > 0 {
}

You don't use a read just for checking for input, there's no point in that. It only complicates your code, since you end up with that extra line you have to incorporate. You just read and handle the result, which you have to do anyway in one way or another. For example, if you really only expected one line, you'd do something like this:
let mut line = String::new();
if std::io::stdin().read_line(&mut line).unwrap() > 0 { // read_line will return Ok(0) at EOF
    // use line
}

Of course, you'll probably want to handle read errors properly instead of just calling Result::unwrap (even if you ultimately still just bail out, but with a more user-friendly error message).
If your input is line-oriented, you can just use the iterator returned by BufRead::lines. Consuming such an iterator (through a for loop, Iterator::try_for_each or similar) will usually give you the behaviour you want.
Still, using the heuristics like the ones linked to by @BurntSushi before the first read does make sense if you want your program to be robust. These heuristics aim at ensuring stdin is "what you expect" and not something "weird" which may exhibit some unexpected behavior. If the check fails, you probably don't want to read from stdin at all but just pretend that it's already at EOF.
Though I have to admit I myself usually don't include such checks in my programs. Or at least I didn't do so in the past.

The problem I am trying to solve is to prevent reading from stdin when launching my app from a cron when indeed there is nothing in stdin.
my app is a simple s3 uploader that can read from stdin (stream) or just pass the file from the command line (multipart upload):
s3m /path/to/file aws/bucket/file

or pipe something stdin :
 mariabackup --stream=xbstream | s3m aws/bucket/file

But I notice what when running the app from a cron atty returns that there is something to read from stdin and I end trying to read from stdin but since there is no input 0 I end ignoring the arguments (this because of the logic of the application, in where I give preference to stdin, maybe the easy way is just to add a flat to read from stdin)
Because of this (since I may have some input)  if the read length > 0 I could start streaming it, probably there is a way to prevent reading all the time, and find if there is something in stdin (covering cases like when running within a cron) this is what I am trying to figure out, so if have any ideas, hints, please let me know, I would appreciate it

The problem is that your cron implementation is attaching a tty to stdin for some reason, based on what you told us. There is really nothing more to be done here because that is the problem. It's not a problem with this crate.
You're only choice from what I can tell is to either add a flag that forcefully prevents reading from stdin, or somehow modify your heuristic for detecting whether stdin might be readable or not. You haven't really given us much to go on here, other than the fact that your cron daemon is advertising a tty on stdin for subprocesses it executes. (I find that highly unusual. You might consider looking into that issue more deeply, and possibly asking the maintainers why it's happening. Just make sure to provide a minimal reproducible example using libc apis directly.) Otherwise, you could probe at the stdin file descriptor a bit more. For example, what is its file type?

I Will search more about it, how could I test the file type?

https://doc.rust-lang.org/stable/std/fs/struct.FileType.html
Note also the FileTypeExt trait on Unix.


Update doc README

@piiswrong  Can we merge this before mid night so that these changes can be updated to website?


add student: Xu Wang

add student Xu Wang

Good effort but not consistent with what's needed
Should match Rahul's format


feat!: new folder structure

Ref: https://github.com/t3dotgg/rewire-t3-app/pull/2
also restructured internally so the adding are now structured like a mirror of the generated project, instead of being grouped all in one per dep
TODO: Edit https://create.t3.gg/en/folder-structure

Purely out of curiosity, when do you think this proposal might land in main, if at all?

Might push it out during the holidays


Fix: some patches related to error exiting

https://github.com/openssl/openssl/pull/17417 for 3.0

Checklist

[ ] documentation is added or updated
[ ] tests are added or updated

Is any of this applicable to 1.1.1 as well?

Is any of this applicable to 1.1.1 as well?

I checked and found no applicable patch.

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set.  Please review the updates and set the label manually.

Merged, thanks for the backport.


charged_kwh & charged_total are 0

Hi there, I've just installed SmartEVSE and flashed it to version 1.3.8.
Any idea why the charged_kwh & charged_total are 0? They do not return anything.
Thanks!

Do you have a seperate kWh meter to measure EV consumption? If yes, is this by accident a one phase Eastron SDM EV meter? Then it's normal since the one phase and three phase version have different modbus registers.

Hmm. No I didn’t. Is that necessary?
If so, where can I find the compatible meters? I have a three phase
installation.
On Sun, 7 Aug 2022 at 10:26, fluppie @.***> wrote:

Do you have a seperate kWh meter to measure EV consumption? If yes, is
this by accident a one phase Eastron SDM EV meter? Then it's normal since
the one phase and three phase version have different modbus registers.
—
Reply to this email directly, view it on GitHub
https://github.com/serkri/SmartEVSE-3/issues/56#issuecomment-1207357382,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABL5DLHQFLAXR3D5WJN7T5LVX5XNBANCNFSM55XIHM7A
.
You are receiving this because you authored the thread.Message ID:
@.***>

Yes, for example, my car charges with 32A monophase, but the real current is only 30.5A. That's +- 345W difference. Sometimes it's even more. Most of the time it charges 6950-7050W instead of the 7400W that's promised. Tesla model 3 really charges on 7400W on this charger. So that's why a meter is needed. Best is to take a MID certified one for accuracy and if you want/need to get kWh's back from your employer or company if the meter is on a private person.
Myself I like Eastron meters, you could take a cheap option SDM72 v2 or a more expensive SDM630.
I ordered a time or 5 from this webshop: https://stromzähler.eu/stromzaehler/drehstromzaehler/fuer-hutschiene-geeicht/246/sdm72dm-v2-mid-3-phasen-zweirichtungs-drehstromzaehler-mit-rs485-und-s0

Thanks. I’ll order one of those!
On Sun, 7 Aug 2022 at 14:06, fluppie @.***> wrote:

Yes, for example, my car charges with 32A monophase, but the real current
is only 30.5A. That's +- 345W difference. Sometimes it's even more. Most of
the time it charges 6950-7050W instead of the 7400W that's promised. Tesla
model 3 really charges on 7400W on this charger. So that's why a meter is
needed. Best is to take a MID certified one for accuracy and if you
want/need to get kWh's back from your employer or company if the meter is
on a private person.
Myself I like Eastron meters, you could take a cheap option SDM72 v2 or a
more expensive SDM630.
I ordered a time or 5 from this webshop:
https://stromzähler.eu/stromzaehler/drehstromzaehler/fuer-hutschiene-geeicht/246/sdm72dm-v2-mid-3-phasen-zweirichtungs-drehstromzaehler-mit-rs485-und-s0
—
Reply to this email directly, view it on GitHub
https://github.com/serkri/SmartEVSE-3/issues/56#issuecomment-1207393487,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABL5DLANROC62XHSRZHVSITVX6RDZANCNFSM55XIHM7A
.
You are receiving this because you authored the thread.Message ID:
@.***>

How does ev_meter works when you have it set to API. can you send the data through the API, rather than through RS485? Cant find any documentation on that.
Thanks for the great work, found this branch today after trying to get the data into Home Assistant from the original mstegen branch, without any success.

I have no clue :-) but I don't think that it works for EV Meter? I think only for Mainsmeter, but maybe @k-janssens can tell you that?
https://swagger-ui.serkri.be/#/Operations/updateCurrentsData here is the example for updating Mainsmeter data.
Myself I really needed the separate EV kWh meter to have home charged kWh's reimbursed from my employer. So I did not explore the API variant. But I have to say, the Modbus meters from Eastron are acceptable SDM230 €40 for single phase and SDM72 v2 €66 for 3phase. I would really recommend to pay a little bit extra for MID certified versions.

The mains_meter was not yet in the openapi yaml (I've added it https://swagger-ui.serkri.be/#/Operations/getSettings) although doing a get /settings would reveal it's there (the yaml file needs to be updated manually, this one was forgotten). Anyway, updating the mains_meter is currently only possible via the LCD. When in solar mode, you get the menu options for MainsMeter, when set to API, it's possible to supply the currents by posting them to /updateCurrentsData (see https://swagger-ui.serkri.be/#/Operations/updateCurrentsData). This indeed bypasses the RS485.


Change in class binding breaks rendering

When a calendar has a class binding with for example :class="{'disabled': !isActive}", and the isActive property changes, the changing class causes the rendering of the calendar to break.
See this codepen for a demo of the issue(click the button in the top left):
https://codesandbox.io/embed/fullcalendar-vue-1lktx

Hmm, that is frustrating... I will try and have a look this afternoon.

I'm guessing this is a conflict between Vue and Fullcalendar's jQuery dom manipulation.
Could you apply the class to a parent element?

Luckily, I can (and I did) use that workaround. I posted because I thought you (and others) should know. Hopefully it's fixable. Thanks for looking into the issue!

Ah ok, glad that solved it. Yeah I guess we could wrap the calendar target element with a parent to prevent this. Not sure whether it would break other's implementations though. One to think about going forward I think.


getJavascriptErrors doesn't return errors

    open("http://the-internet.herokuapp.com/javascript_error");
    List<String> jsErrors = getJavascriptErrors();
    System.out.println(jsErrors);
    assertNoJavascriptErrors();

This should return one JavaScript error (Cannot read property 'xyz' of undefined), but it doesn't (Firefox 45)

Yes, method getJavascriptErrors has some restrictions. Namely, it only catches JS errors that happens AFTER page loading. Unfortunately it's not technically possible to catch JS errors during page loading with Selenium.
As far as I know, the only way to do it is to use proxy server and inject custom JS code into the server HTML response.


Failed to import enable-check

Using this library with Nuxt.js will cause a fatal error.
Package subpath './enable-check' is not defined by "exports"

I tried to import vue-tsx-support/enable-check into nuxt.config.ts.
But when I start up the development environment, I get this error.
This reproduction is here: https://codesandbox.io/s/vue-tsx-support-repro-import-issue-4o81s

You need to add "vue-tsx-support/enable-check.d.ts" to the types array of your tsconfig

Thanks for the response.
I have tried the two patterns below. But it is not solved.

Add enable-check.d.ts path to the types array of tsconfig. (I got the same error as in the first comment.)
In Additional to the first pattern, remove the enable-check import from nuxt.config.ts. (I was able to launch nuxt, but I get a type error that cannot interpret the built-in HTML tags. Reproduction is here)

I think this error is occurred because enable-check.js is not exported in the package.json.
https://github.com/wonderful-panda/vue-tsx-support/blob/v3.2.0/package.json#L8-L19


Role does not exist

When I changed the port on entity-services-examples/gradle.properties to 8204, 'gradle mlDeploy' gave the following:
$ gradle mlDeploy
:mlDeleteModuleTimestampsFile
:mlPrepareRestApiDependencies
:mlDeployApp
:mlPostDeploy UP-TO-DATE
:mlDeploy
:entity-services:mlDeleteModuleTimestampsFile
:entity-services:mlPrepareRestApiDependencies
:entity-services:mlDeployApp
:entity-services:mlPostDeploy UP-TO-DATE
:entity-services:mlDeploy
:entity-services-e2e:mlDeleteModuleTimestampsFile
:entity-services-e2e:mlPrepareRestApiDependencies
:entity-services-e2e:mlDeployApp
:entity-services-e2e:mlPostDeploy UP-TO-DATE
:entity-services-e2e:mlDeploy
:entity-services-examples:mlDeleteModuleTimestampsFile
:entity-services-examples:mlPrepareRestApiDependencies
:entity-services-examples:mlDeployApp FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':entity-services-examples:mlDeployApp'.
> Unable to insert content at URI: /ext/Race-0.0.1-source-2.xqy; cause: Role does not exist

You have to run the examples from the examples directory.  I use the gradle wrapper.
cd entity-services-examples
../gradlew mlDeploy

This was an EA-3 decision because of some underlying issue with paths and ml-app-deployer

I'll be updating the readme to include in most recent PR.

Ok, I'll just close this and I won't try to run it from the root folder.


Don't allow comment when locked

Tweaking b/c you can comment on your own stuff even if locked. Hmm. viewerCanUpdate is for editing...
The Issue API doesn't have a viewerCanComment or anything.
@Sherlouk can you use the API explorer and see if you can administer this repo?
query {
  repository(owner: "rnystrom", name: "githawk") {
    viewerCanAdminister
  }
}

"viewerCanAdminister": false 😢

Crap! Wtf. How are we able to tell who has commenting permissions while locked?
@Sherlouk are you able to lock issues on this repo?

Yup!

Grrr this is frustrating.

viewerCannotUpdateReasons references the issue being locked, we sure viewerCanUpdate doesn't have a double purpose?

@Sherlouk ah wait, I think you're right. I thought that I owned the playground repo, but I made it w/ my test account. I think viewerCanUpdate is ok then.

Hmm no I don't think it is viewerCanUpdate is false for my alt account on this issue - that's checking whether I can update the root message

Collaborator:
"viewerCanAdminister": false,
"viewerCanSubscribe": true,
"viewerCanUpdateTopics": false,
"viewerCanCreateProjects": true
Normal User:
"viewerCanAdminister": false,
"viewerCanSubscribe": true,
"viewerCanUpdateTopics": false,
"viewerCanCreateProjects": false

Also we need to be careful because GitHub treats permissions differently for repos owned by an organisation versus repos owned by a "normal user"
https://help.github.com/articles/access-permissions-on-github/

Hopefully the “viewerCan” fields are enough info. Wish the field to actions mapping was more clear.
Sent with GitHawk

Ugh, this is messy :(

Maybe we just use hidden = locked && !viewerCanAdminister? Non-admin collabs wont be able to comment, but IMO its weird that we all can comment when its locked, lol.

I don't see locked conversations enough to see why people use them honestly - but I feel we should try n match GH as much as possible despite how piss poor the API is in places

Eg, Fastlane is doing auto closing and later locking. Prevents users from adding comments on a (possibly) outdated issue, forcing them to create a new one. Could go well with #405 maybe?!

Wrote a question on the API forums for this
https://platform.github.community/t/which-permissions-to-use-for-issue-actions-lock-label-close/3390

Alright got a v1 of this working


Feature: Redesign the LC provider interface to a uniform REST interface

For phase I , the LC manager is using a specific local instance of the provider client in order to communicate with the provider. This make the LC manager aware of the concrete implementation of the LC provider. This creates dependency between the provider implementation and the LC manager. Also this require the LC manager to configure the provider.
The idea is to convert the provider interface into a pre-define set of REST APIs, so that connecting into a provider just require regular authentication without instantiating a local instance of the specific provider client code.

placing in sprint 3

streaming of watch events is a current design consideration

Update Aug 24: Still WIP. This is not something that's critical to sprint 3 and we may push it to 4.

not  relevant for ks-0.20


mongo-optimization task Slava.telyatko

https://travis-ci.org/github/SlavaTel/db-assignments

https://travis-ci.org/github/SlavaTel/db-assignments

Closing the PR as accepted! Thanks!


Make sure there is a suspicious code warning for blocks in => functions that look like objlits.

See https://twitter.com/aboodman/status/633724451020890112
["foo"].map(s => {key:s}) kind of looks like it's returning an object literal {key:s} but actually (spoiler alert!) the {}'s are parsed as a block, so this is equivalent to
["foo"].map(function(s) {
  key:
    s;
}

where key is a label and s is a useless EXPR_RESULT. We should make sure we emit some kind of warning for this case. It looks like we don't currently: http://closure-compiler-debugger.appspot.com/#input0%3D%255B%2522foo%2522%255D.map(s%2520%253D%253E%2520%257Bkey%253As%257D)%26input1%26conformanceConfig%26externs%26refasterjs-template%26includeDefaultExterns%3D1%26CHECK_SUSPICIOUS_CODE%3D1%26CHECK_SYMBOLS%3D1%26CHECK_TYPES%3D1%26LANG_IN_IS_ES6%3D1%26MISSING_PROPERTIES%3D1

Great idea. Two specific checks I can think of are:

Array.map() with a function that doesn't return anything.
Arrow functions using the block syntax where the block also parses as an object literal.

I think an unused label check would catch these kinds of these problems everywhere without special casing places they are allowed.


Add CoC and references to it

Just copied example CoC from https://www.contributor-covenant.org/version/2/0/code_of_conduct/
We can use another one or adapt as needed.

I think that to merge this one would be nice to get at least 2 or maybe 3 approvals, just cause it mentions policies to ban people and this kind of stuff. In general I personally agree with everything in this CoC, let me know what you guys think.

Or maybe we could just reuse kubernetes CoC as well, whatever works :)


Refactor Delete

#4460

@tiberiuichim it would be great if we could have your opinion on https://github.com/plone/volto/pull/4890/commits/22a850eaaca3dcce696b776f884aea53d33a75c3 too.

@sneridagh @JeffersonBledsoe My initial gut reaction is that I don't see the benefit of creating individual hooks like useContent, useDeleteContent, etc. It feels like boilerplate that just adds another layer of indirection, and I don't really understand how it would help with moving to useQuery/useMutation in the future.

I would vote for creating a single trigger hook for all API calls that accepts a callback for each action. Though I don't know if it should come with the refactoring process or be the thing of future.

@davisagli @tiberiuichim @nileshgulia1 @JeffersonBledsoe I'm quite torn on this one.
The initial idea for the hooks façade was from @tiberiuichim who mention it at some point in one of our countless discussions and musing on Volto things. I thought that it was the right moment to implement it since it would allow us to a swift and soft transition when we move to the new data fetching architecture.
Again, the idea is that if the façade exists, then the changes to the components when the new data fetching library is ready, will be as easy as make an add-on that shadows all the façade and replace them with the RQ ones. No other intervention required.
This is, of course, in theory... coming up with a 1:1 API architecture for the façade is complex. But on the paper, it should work. However, this might also change if Suspense is around (then the way of writing the components will also be different), and we will have to face a component change again... 😫
Maybe the reality is that it's too overkill... and that the React ecosystem will force us to rewrite the components again, so the effort is futile.  Coming up with a sane API is complex (as expected), and not without doubts.
<aybe we will have to live with the idea that when we made the change, the components should be also changed (maybe using the @plone/components ones)...
I'd like to know your opinion as well...
/cc @plone/volto-team

I'd like to get to an agreement soon. I plan to discuss about it tomorrow during the Volto Team Meeting. This is right now a blocker for @Tishasoumya-02 to continue with the project.
/cc @plone/volto-team


fix: spacings in tutorial

FX-4261
Description
⚠ Changes to stories are updated in https://github.com/toptal/picasso/pull/3870
This pull request updates the How to use spacings tutorial to depict the latest changes in spacing.
How to test

Temploy
See the https://picasso.toptal.net/fx-4261-update-guidelines-on-spacing-usage-3/?path=/story/tutorials-how-to-use-spacings--how-to-use-spacings

Screenshots

Development checks

[N/A] Add changeset according to guidelines (if needed)
[x] Read CONTRIBUTING.md and Component API principles
[N/A] Make sure that additions and changes on the design follow Toptal's BASE design, and it's been already discussed with designers at #-base-core
[N/A] Annotate all props in component with documentation
[N/A] Create examples for component
[N/A] Ensure that deployed demo has expected results and good examples
[N/A] Ensure the changed/created components have not caused accessibility issues. How to use accessibility plugin in storybook.
[x] Self reviewed
[N/A] Covered with tests (visual tests included)

All development checks should be done and set checked to pass the
GitHub Bot: TODOLess action

PR commands

List of available commands:

@toptal-bot run package:alpha-release - Release alpha version
@toptal-anvil ping reviewers - Ping FX team for review

PR Review Guidelines

When to approve? ✅
You are OK with merging this PR and

You have no extra requests.
You have optional requests.

Add nit: to your comment. (ex. nit: I'd rename this variable from makeCircle to getCircle)

When to request changes? ❌
You are not OK with merging this PR because

Something is broken after the changes.
Acceptance criteria is not reached.
Code is dirty.

When to comment (neither ✅ nor ❌)
You want your comments to be addressed before merging this PR in cases like:

There are leftovers like unnecessary logs, comments, etc.
You have an opinionated comment regarding the code that requires a discussion.
You have questions.

How to handle the comments?

An owner of a comment is the only one who can resolve it.
An owner of a comment must resolve it when it's addressed.
A PR owner must reply with ✅ when a comment is addressed.

@toptal-anvil ping reviewers


[5.9] Minimized mode bottom margin should match side padding

Explanation: Quick Nav bottom margin should also increase when font size increases
Scope: Quick Nav only
Issue: rdar://108577661
Risk: Small
Testing: Manual testing
Reviewer: @mportiz08
Original PR: #617

@swift-ci test


Allow SideMenu to be called from tabbar controller

This implements the solution proposed by @albernackee in #8 Currently if you try to try to present the sidemenu from a scene embedded in a tab bar controller you will receive a warning saying that the navigation controller is nil. This fix makes the presenting view controller not be the tab bar itself, but the controller embedded inside of the nav controller.
Feeback is appreciated

Thank you for contributing, @saltyskip. I simplified your logic in case you're interested to see the result.

Awesome I scratched my head for a bit trying to make it cleaner, but gave up. Good to see a better version! @jonkykong


Explain that thumbprint is not part of x 509 spec

It would be useful to mention that the thunbprint is calculated by the Windows tools and is not part of the certificate specification and you will not see a thumbprint  field in a cert when you view it with other tools like OpenSSL.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ba798f60-80a8-5d2c-555c-dd62917111ec
Version Independent ID: b3e6d006-b2e4-3959-46f6-e5a6284ee3d4
Content: How to: Retrieve the Thumbprint of a Certificate
Content Source: docs/framework/wcf/feature-details/how-to-retrieve-the-thumbprint-of-a-certificate.md
Product: dotnet-framework
GitHub Login: @Lxiamail
Microsoft Alias: wcfsrvt

The statement about removing the spaces in between the hex numbers was very valuable info. Thank you!

Is there a way I can access the thumbprint from powershell/cmdline?


fix: remove vite-ignore attribute error

Description
fix #19055

Do we need to update this as well?
https://github.com/vitejs/vite/blob/eb06ec30bb02ced66274f0fc6e90aff2bb20c632/packages/vite/src/node/plugins/html.ts#L454

Do we need to update this as well?
https://github.com/vitejs/vite/blob/eb06ec30bb02ced66274f0fc6e90aff2bb20c632/packages/vite/src/node/plugins/html.ts#L454

There should be no need to modify this, it is normal.

There should be no need to modify this, it is normal.

Ah, I somehow read it wrong 🤦 Please forget about it.


Xml Substitution does not work if value is empty string

Environment

Server - VSTS

Issue Description
This used to work well in the past, but it has stopped working since last two days.
Consider web.config
<appSettings>
     <add name="CDNHost" value="" />
     <add name="SignalRHost" value="localhost" />
</appSettings>

Substitution
CDNHost = "some-cdn.azureedge.net"
SignalRHost = "notify.some-host.com"
Expected output
<appSettings>
     <add name="CDNHost" value="cdn.azureedge.net" />
     <add name="SignalRHost" value="notify.some-host.com" />
</appSettings>

Actual Output
<appSettings>
     <add name="CDNHost" value="" />
     <add name="SignalRHost" value="notify.some-host.com" />
</appSettings>

Notice that CDNHost remains empty. This was working correctly two days ago.

@neurospeech can you specify the task version that the release is picking up ?

IIS Web App Deploy Version is [0.*]
[AGENT_VERSION] --> [2.126.0]
[AGENT_WORKFOLDER] --> [C:\vstsagent\A1\_work]
[AZURE_HTTP_USER_AGENT] --> [VSTS_ad14dd5b-81d8-4ee5-af36-fe96e7927997_release_2_404_640_1]

Since the log contains sensitive information, I will need to know what fields you need from the release log.

Please specify the exact task version from the release logs.
2018-02-13T09:57:02.3630259Z ==============================================================================
2018-02-13T09:57:02.3630491Z Task         : IIS Web App Deploy
2018-02-13T09:57:02.3630652Z Description  : Deploy a Website or Web Application using WebDeploy
2018-02-13T09:57:02.3630831Z Version      : 0.0.34
2018-02-13T09:57:02.3631013Z Author       : Microsoft Corporation
2018-02-13T09:57:02.3631189Z Help         : More Information
2018-02-13T09:57:02.3631388Z ==============================================================================

2018-02-13T08:57:28.6464653Z ##[section]Starting: IIS Web App Deploy
2018-02-13T08:57:28.6468542Z ==============================================================================
2018-02-13T08:57:28.6468643Z Task         : IIS Web App Deploy
2018-02-13T08:57:28.6468720Z Description  : Deploy a Website or Web Application using WebDeploy
2018-02-13T08:57:28.6468797Z Version      : 0.0.30
2018-02-13T08:57:28.6468880Z Author       : Microsoft Corporation
2018-02-13T08:57:28.6468957Z Help         : [More Information](https://aka.ms/iiswebappdeployreadme)
2018-02-13T08:57:28.6469039Z ==============================================================================
2018-02-13T08:57:57.0118336Z XML variable substitution applied successfully.

@neurospeech we have fixed this issue in version 0.0.33 of the task which is currently being deployed. Depending on the location of your account, you will receive this fix within two weeks.

I can confirm this bug exists for the App Service deploy task version 3.3.36.  Is the fix for that task currently being deployed as well?

@NateB2 , the issue is fixed in 3.3.40 task version. The fix will be available in 2-3 weeks.

@NateB2 , The Fix is deployed. Rerun the release for the new task version.

Hello @vincentdass can you tell me if the other direction of empty string substitution shall work?
I try to replace a value in my XML config with an empty string (modified initial example for clarification).
Consider web.config
<appSettings>
     <add name="CDNHost" value="abc" />
     <add name="SignalRHost" value="localhost" />
</appSettings>

Substitution
CDNHost = "" empty string
SignalRHost = "notify.some-host.com"
Expected output
<appSettings>
     <add name="CDNHost" value="" />
     <add name="SignalRHost" value="notify.some-host.com" />
</appSettings>

Actual Output
<appSettings>
     <add name="CDNHost" value="abc" />
     <add name="SignalRHost" value="notify.some-host.com" />
</appSettings>

Notice that CDNHost still has the original value.
I'm using this version of the  Task:
2018-03-02T10:58:28.1088669Z ==============================================================================
2018-03-02T10:58:28.1088669Z Task         : Azure App Service Deploy
2018-03-02T10:58:28.1088669Z Description  : Update Azure WebApp Services On Windows, Web App On Linux with built-in images or docker containers, ASP.NET, .NET Core, PHP, Python or Node based Web applications, Function Apps, Mobile Apps, Api applications, Web Jobs using Web Deploy / Kudu REST APIs
2018-03-02T10:58:28.1088669Z Version      : 3.3.41
2018-03-02T10:58:28.1088669Z Author       : Microsoft Corporation
2018-03-02T10:58:28.1088669Z Help         : [More Information](https://aka.ms/azurermwebdeployreadme)
2018-03-02T10:58:28.1088669Z ==============================================================================

@Andreas-Lehmann FYI, I hit the same problem as you so I opened #8631.


About the code to render images from point cloud for objaverse

Hi,
It is a great job. I am really interested in it. Could you provide the code of rendering 12 images from point cloud for objaverse.
Thanks!

We render the mesh instead of the point cloud. Please refer to the code


If you have multiple Chrome windows (not tabs), the PointedWindow definition will be executed in the ForegroundWindow.

ver 4.19.456.0
1.Start chrome and create multiple tabs.
2.Separate the tabs into separate windows.
3.Hover over an inactive chrome window and try to perform the gesture defined in the PointedWindow.
4.The process defined in the active chrome Window is executed.
-> I want the process defined on the inactive chrome Window to be executed.

Hi @twinbird827 , thank you for using crevice!
When there are many windows of the app,  even if you started the gesture on the background one, SendInput will send the message to the foreground window. So, if you want to change the target to PointedWindow, you have to activate it explicitly.
Do(ctx =>
{
    ctx.PointedWindow.Activate(); // bring the PointedWindow to the top
    SendInput.Multiple().

thank you!
It works as expected.


Remove deprecated viewer module

Codecov Report

:exclamation: No coverage uploaded for pull request base (v4@57b634d). Click here to learn what that means.
The diff coverage is n/a.

:exclamation: Current head 917b82e differs from pull request most recent head 4d519a7. Consider uploading reports for the commit 4d519a7 to get more accurate results

@@          Coverage Diff           @@
##             v4     #1032   +/-   ##
======================================
  Coverage      ?   100.00%
======================================
  Files         ?        35
  Lines         ?      2457
  Branches      ?         0
======================================
  Hits          ?      2457
  Misses        ?         0
  Partials      ?         0


Update 4.0 yaml to work out of the box

Description

Update Makefile to make it easier for new developers
Create develpoment docs
Update manager.yaml with custom grafana-operator env vars
to work out of the box

Relevant issues/tickets

#447

Type of change

[X] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

[ ] This change requires a documentation update
[X] I have added tests that prove my fix is effective or that my feature works
[ ] I have added a test case that will be used to verify my changes
[ ] Verified independently on a cluster by reviewer

Verification steps

I did the correct thing and broke out the PR in to two PR so now the docs is on it's own and the yaml fix on it's own.
https://github.com/grafana-operator/grafana-operator/pull/456

POD_NAME and OPERATOR_NAME have been deprecated.
https://sdk.operatorframework.io/docs/building-operators/golang/migration/
"OPERATOR_NAME and POD_NAME environment variables are no longer used"
I can't see anything about TEMPLATE_PATH in the code. It looks like it was used some way in https://github.com/integr8ly/integreatly-operator
I'm unsure if this is needed any more, I don't think so unless it's used in the background with the integreatly-operator and that's why we should keep:
          - name: TEMPLATE_PATH
            value: /usr/local/bin/templates

that is perfect, thanks @NissesSenap
The template path is a relic from the pre 3.x days. Should be safe to remove it.


Problem to run  in AOT mode using SystemJS loader

I'm submitting a ...
[x] Bug report
Current behavior No. 1
Using UMD bundle angular-modal-gallery.umd.js
Compiling an app via Angular ngc compiler and running the app with SystemJS loader in AOT mode platformBrowser().bootstrapModuleFactory(AppModuleNgFactory) and with the following systemjs.config.js leads to the situation when http://localhost:8080/node_modules/angular-modal-gallery/angular-modal-gallery.ngfactory.js file is requested to load and it is not included in UMD bundle (because of the app's AOT mode):
map: {
     ...,
     'angular-modal-gallery': './node_modules/angular-modal-gallery/bundles'
},
packages: {
     ...,
     'angular-modal-gallery': {main: 'angular-modal-gallery.umd.js'}
}

Current behavior No. 2
Using directly angular-modal-gallery.ngfactory.js
The systemjs.config.js is a bit different and files are loaded directly ... first angular-modal-gallery.ngfactory.js is loaded without any problem but there is a problem with loading the related angular-modal-gallery.js because a wrong request is sent to a server ... http://localhost:8080/node_modules/angular-modal-gallery and the server returns index.html code instead.
The same logic is possible to use without any problem for example with the latest PrimeNG components (each component is located in a separate ng module).
map: {
     ...,
     'angular-modal-gallery': './node_modules/angular-modal-gallery'
},
packages: {
     ...,
     'angular-modal-gallery': {defaultExtension: 'js'}
}

Expected behavior
Able to use angular-modal-gallery component in AOT mode.
Environment
- Node version: any
- Platform:  any
- Angular version: 5.1.0

Browser: any

I apologize ... I forgot that the configuration as following solves my situation ... just using UMD bundle angular-modal-gallery.umd.js together with possibility to load angular-modal-gallery.ngfactory.js:
map: {
     ...,
     'angular-modal-gallery': './node_modules/angular-modal-gallery'
},
packages: {
     ...,
     'angular-modal-gallery': {
          main: 'bundles/angular-modal-gallery.umd.js',
          defaultExtension: 'js'
     }
}


feat: Add OneDrive Integration

:tada: This PR is included in version 1.37.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Training fails with JSONDecodeError

This is with CUDA on the cluster, command snakemake -p --profile config/slurm/hemera imagenette2_train. Snakemake itself fails with an error. A bunch of jobs start, then this happens:
Traceback (most recent call last):
  File "/home/pape58/Code/sota_on_uncertainties/venv/lib/python3.9/site-packages/snakemake/__init__.py", line 722, in snakemake
    success = workflow.execute(
  File "/home/pape58/Code/sota_on_uncertainties/venv/lib/python3.9/site-packages/snakemake/workflow.py", line 1110, in execute
    success = self.scheduler.schedule()
  File "/home/pape58/Code/sota_on_uncertainties/venv/lib/python3.9/site-packages/snakemake/scheduler.py", line 421, in schedule
    self._finish_jobs()
  File "/home/pape58/Code/sota_on_uncertainties/venv/lib/python3.9/site-packages/snakemake/scheduler.py", line 524, in _finish_jobs
    self.get_executor(job).handle_job_success(job)
  File "/home/pape58/Code/sota_on_uncertainties/venv/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 875, in handle_job_success
    super().handle_job_success(
  File "/home/pape58/Code/sota_on_uncertainties/venv/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 232, in handle_job_success
    job.postprocess(
  File "/home/pape58/Code/sota_on_uncertainties/venv/lib/python3.9/site-packages/snakemake/jobs.py", line 1091, in postprocess
    self.dag.workflow.persistence.finished(
  File "/home/pape58/Code/sota_on_uncertainties/venv/lib/python3.9/site-packages/snakemake/persistence.py", line 239, in finished
    starttime = self._read_record(self._metadata_path, f).get(
  File "/home/pape58/Code/sota_on_uncertainties/venv/lib/python3.9/site-packages/snakemake/persistence.py", line 423, in _read_record_uncached
    return json.load(f)
  File "/trinity/shared/pkg/devel/python/3.9.6/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/trinity/shared/pkg/devel/python/3.9.6/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/trinity/shared/pkg/devel/python/3.9.6/lib/python3.9/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 1164 (char 1163)

The output right before that (though I'm not sure if it is related or if the results of some other job were collected at that point):
mkdir -p data/imagenette2-320-all/n01440764 && cp -vur data/imagenette2-320/val/n01440764/*JPEG data/imagenette2-320/train/n01440764/*JPEG data/imagenette2-320-all/n01440764
Submitted job 5 with external jobid 'Submitted batch job 4797615'.

This is something totally unexpected.

This seems to have solved it. Thank you 👍🏻


GetFolderAsync & GetFileAsync error handling problem

These methods do not return a null if they encounter an error. I have a coroutine that is called by an XAML button click handler. This coroutine contains both of these methods. When an error occurs try/catch does not work instead the behavior is as if the handler had completed bypassing any recovery code.
In the future, will these methods be changed to return a null ?

These methods throw an exception on failure. You can catch the exception within the coroutine as follows:
StorageFolder folder = KnownFolders::DocumentsLibrary();

try
{
    StorageFile file = co_await folder.GetFileAsync(L"filename.txt");
}
catch (hresult_error const& e)
{
    WINRT_TRACE("%ls\n", e.message().c_str());
    throw;
}

I'm using
catch (hresult_error const & e)
in the try/catch.
Could the problem be that it's in button click handler ?

That shouldn't be an issue.

The plot thickens...
I'm using a copy of iBuffer for my test code. I created a new method that includes my test code. I call this method in wWinMain before Application::Start,,, The try/catch in this new method (with my code) WORKS  as advertised.
But that doesn't solve my problem because I need it in an XAML button click.

Ah, there is a difference!!!
My problem coroutine, that is in the button click.  contains 3 parameters. Each of which is a class being passed by reference.. These classes are not referenced until AFTER the get folder and get file. My initial tests indicate that files reads are able to update these classes.
The problem appears to be that the mere presence of such parameters causes any try/catch to fail.
My workaround is 2 coroutines; 1 with no parameters  that checks are corrects missing  folders and files and a second with parameters. which update the classses.

You should avoid reference parameters on coroutines involving suspension points (async calls).

The purpose of my coroutine is to replicate my C#  UWP App's deserialization method. This method is called only in the app class's constructor and app class  OnLaunched method. This is occurs while the app;s logo is displaying and prior to the call rootFrame.Navigate(typeof(MainPage), e.Arguments);..
That being said, is it safe to use reference parameters or should I use out parameters.
In any case what is the proper way to handle serialization and  deserialization in a C++/winRT app? (stay  with C# is not an option).

I'm closing this and will attempt to use global classes instead. If that doesn't work I create a new issue._


T1060 create links data

First step in moving logic concerning online links out of item.html and into voyager.py. Added function to voyager.py to create new ONLINE list of holdings data from marc856list and ELECTRONIC_DATA.
Some minor ui improvements, including using a blue button for anything available to GW community even if catalog record from other campus. Removed [ ] brackets around labels.
Passes launchpad-autotest, but should still get a lot of code review and testing.
Fixes #1060. See #1060 comments for next steps.

Also fixes #1065

Reassigning this back to me to deal with the duplicate GT links as seen in bibid 12019336.


:bug: fix for sending b3 traces and python apm package update

Changes
Please provide a brief description of the changes here.
Merge Requirements
For new features contributions please make sure you have completed the following
essential items:

[ ] CHANGELOG.md updated to document new feature additions
[ ] Appropriate documentation updates in the docs
[ ] Appropriate Helm chart updates in the helm-charts

Maintainers will not merge until the above have been completed. If you're unsure
which docs need to be changed ping the
@open-telemetry/demo-approvers.

LGTM


chore(main): release 1.1.50

:robot: I have created a release beep boop
1.1.50 (2023-09-04)
Bug Fixes

deps: update dependency @tailwindcss/forms to v0.5.5 (18ad385)
deps: update dependency @tailwindcss/forms to v0.5.6 (19bcae0)
deps: update dependency ai to v2.2.10 (a543fd9)
deps: update dependency ai to v2.2.11 (5bee61f)
deps: update dependency ai to v2.2.7 (e08b410)
deps: update dependency ai to v2.2.8 (db0a1f2)
deps: update dependency ai to v2.2.9 (dd9dca6)
deps: update dependency chart.js to v4.4.0 (4f408bf)
deps: update dependency postcss to v8.4.29 (eb86262)
deps: update dependency typescript to v5.2.2 (ec98439)
deps: update prisma monorepo to v5.2.0 (7a3d33d)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/icoretech/airbroke/releases/tag/v1.1.50 :sunflower:


Add a codec and interface registry to E2ETestSuite

Summary
When writing e2e tests we may need access to a codec and interface registry in order to marshal/unmarshal various types.
Currently we create one by hand, for example when handling genesis modification pre-startup.
It would be useful to have access to this via the E2ETestSuite in some capacity.

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged/assigned

Implemented https://github.com/cosmos/ibc-go/blob/main/e2e/testsuite/codec.go


prepare builder for zola

https://github.com/getzola/zola

commit :
https://github.com/threefoldtech/jumpscaleX_builders/commit/794fd8e833543fb4b2bdbdfcfe25ab665f147f12
Added zola builder.

verified.


Windows binary works incorrect

It creates files on remote side with wrong slashes:

What exactly is wrong with this? Isn't windows using \ instead of the usual /

It is Linux on screenshot, so Spot cannot run script on remote side due to wrong filename.
Windows, btw, support both of slashes.

So, this is the case with the host running spot is windows but the destination nodes are linux. In this case, I do understand what's going on - the host doesn't have a clue what the destination machine is and making tmp path with filepath.Join which leads to this.
I think the simplest solution is to enforce linux / regardless of what host OS is

pls check if the latest 1.10.0 solved the issue


Replace and join expressions return null when the match not found

Hi I have a variable,
images: "genevafluentd_td-agent,genevamdm,kured". I am trying to use each expression on jobs
jobs:
- ${{ each image in split(variables.images, ',')}}:
#- job: ACR_${{ replace(variables.image,'-','') }}Replication
- job: ACR${{ join('', split(variables.image,'-')) }}_Replication
displayName: ${{ image }}_ImageReplication
It works as expected when the image name is: "genevafluentd_td-agent". When it is iterating over "genevamdm" and "kured" values, replace and join returns null.

@hima-gadupudi Does this issue refer to a specific article?