"Skipping standard library file" notification spam whenever a source file is open

Skipping standard library file (stdlib excluded): c:\Program Files\Python39\Lib\typing.py

Skipping standard library file (stdlib excluded): e:\Users\Avasam\Documents\Git\typeshed\.venv\Lib\site-packages\openpyxl\descriptors\serialisable.py

Skipping standard library file (stdlib excluded): e:\Users\Avasam\Documents\Git\typeshed\.venv\Lib\site-packages\openpyxl\descriptors\base.py

...

Almost everytime I save a file. (so whenever mypy runs/updates I guess)
This makes working on stubs quite annoying. I'm not sur of the value of this warning for regular users, but is there a way for stubs developper to ignore it? Or even to no skip "standard library file" (except typing.py those are not stdlib and getting mypy hints is still useful)

@Avasam You might have enabled mypy-type-checker.showNotification to warning. You can wither turn it off by removing the setting or you can reduce the notification to error

Even worst, it was on "always". Thanks!


why use vgg rather than resnet?

i am a bigginer, i want kwoun the reason use vgg , or resnet is ok also.

The VGG part is encoding the features only, so the difference between the two here (by your knowledge of classifiers I believe) is that resnets have skip connections all the way, while VGG is pure conv blocks without skips.
They use the VGG like convolutions to encode information, and by the size of their VGG theres no real reason to implement skip connections I believe (they dont use the whole 16 layers, only 6).
Also, they do use skip connections on their decoder process(5 of the 6 convs have skip connections to the upsample + conv block).
I'm not an expert to say why the choose of VGG instead of ResNet, but by what I've worked, usually adding skip connections to this convolutions don't make any improvement, and sometimes its even much worse than not having it, I have a feeling that if you want to encode information as best as you can in a smaller vector space, skip connections on short nets like such don't help at all.

I would also remind you that ResNets use skip connections via addition (and not concatenation like the unet), so the idea of using the skip connections are purely to preserve the gradient at early layers, and with that, be allowed to create deeper models. 6 layers wont have any gradient problem.


Add minification to build process.

Hi @zzarcon, please take a look when you have some time :)

resolves #9

Really good one! I think the final user will really appreciate to have both, min and unminfied dists  :+1:


Additional dash in argument parser for iterative refinement generator

In this line, group.add_argument('---iter-decode-eos-penalty', ...) contains an extra dash.
I will make a PR to fix this soon.

Closed by this commit


Makijaku(Japanese word) for sewing mesuring tape...i guess i am old. :-P

Makijaku(Japanese word) for sewing mesuring tape...i guess i am old. :-P created by<EMAIL_ADDRESS>on 2016-12-23 11:34:10

<EMAIL_ADDRESS>replied, Anyone else confirm please?

<EMAIL_ADDRESS>replied, im sure you all have heard this...Belauan acccent---makishak or makisiak :-P


Add menu prop to Toolbar

From the MDC Toolbar docs:

There are two types of icons, mdc-toolbar__icon--menu represents the left most icon in mdc-toolbar usually to the left of mdc-toolbar__title

This will allow fixing the left/right padding on toolbar icons. I'm working on updating the demo/docs site. (AFAIK a release has to be made before I can compile the docs site?)

Seems to work, but as we'd like to have the exact same output as material-components-web we would prefer to not have mdc-toolbar__icon when mdc-toolbar__icon--menu is used.
So maybe it would be better to make a Toolbar.MenuIcon component (which seems kind of overkill) or to set the className based on whether or not the menu prop is used.

Ah, so with the current way, it would generate both class names. I'll implement the second solution (for now) - unless there's precedent in another component for this situation?

I guess a new component would be great, check if u can figure a way to use icon. Also please check if mdc-web allows to change the icon itself from the hamburger menu. In this case lets take the content of icon from a prop or markup

MDC-Web doesn't really limit what to put inside an element with that class. For what it's worth, in the MDC demos for Toolbar, they conform to how we're generating it right now.
I don't think Icon would work as it uses <i class="material-icons">, while toolbar icons use <a class="material-icons"> (at least in the mdc demos).


Add system for flares to modify powers/spark

Currently setting this up for a work group, noticed that if I add a Flare like Fury, I cannot modify the Flare to add an Effect that modifies an attribute (e.g., +1 Sun) - I have to manually create the passive effect on the Actor to modify their attributes. Would be great if we could just do that within the Flare itself, similar to feats in D&D 5e system.

Progress update


SC-54 Proxy server for testing schema

Coverage decreased (-0.1%) to 70.871% when pulling 2bd91e47c7056ec547e42cec985d928613b271fd on ShwethaKumbla:testSchema into 2997b4aaa7db6544e5bd6e80e7f1396aba78e67a on ServiceComb:master.

Codecov Report

Merging #117 into master will decrease coverage by 0.07%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #117      +/-   ##
==========================================
- Coverage   66.65%   66.58%   -0.08%
==========================================
  Files          18       18
  Lines        3956     3956
==========================================
- Hits         2637     2634       -3
- Misses       1153     1155       +2
- Partials      166      167       +1

Impacted Files
Coverage Δ

server/service/governservice.go
64.07% <0%> (-1.46%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2997b4a...2bd91e4. Read the comment docs.


[Image Request]: SubstrateVM base

What software would you like us to add to our image catalog? (one per issue please)
See https://github.com/GoogleContainerTools/distroless/issues/217#issuecomment-1369358515
which versions of the containerized software do you need?
Libz
Add some justification for why this specific image and versions are important.
No response

cc @tysonkun - we can discuss here. We should be able to get libz into a base image if you can try it out!

Hey @tysonkun - I pushed some test images here:

ghcr.io/dlorenc/images/java-static-amd64
ghcr.io/dlorenc/images/java-static

(these are just tests so I did tags for the different architectures, default is arm64) - they should have zlib and glibc:
% docker run ghcr.io/dlorenc/images/java-static-amd64 ldconfig -p
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
20 libs found in cache `/etc/ld.so.cache'
	libz.so.1 (libc6,x86-64) => /lib/libz.so.1
	libutil.so.1 (libc6,x86-64) => /lib/libutil.so.1
	libthread_db.so.1 (libc6,x86-64) => /lib/libthread_db.so.1
	librt.so.1 (libc6,x86-64) => /lib/librt.so.1
	libresolv.so.2 (libc6,x86-64) => /lib/libresolv.so.2
	libpthread.so.0 (libc6,x86-64) => /lib/libpthread.so.0
	libnss_files.so.2 (libc6,x86-64) => /lib/libnss_files.so.2
	libnss_dns.so.2 (libc6,x86-64) => /lib/libnss_dns.so.2
	libnss_compat.so.2 (libc6,x86-64) => /lib/libnss_compat.so.2
	libnsl.so.1 (libc6,x86-64) => /lib/libnsl.so.1
	libmvec.so.1 (libc6,x86-64) => /lib/libmvec.so.1
	libmemusage.so (libc6,x86-64) => /lib/libmemusage.so
	libm.so.6 (libc6,x86-64) => /lib/libm.so.6
	libdl.so.2 (libc6,x86-64) => /lib/libdl.so.2
	libcrypt.so.1 (libc6,x86-64) => /lib/libcrypt.so.1
	libc_malloc_debug.so.0 (libc6,x86-64) => /lib/libc_malloc_debug.so.0
	libc.so.6 (libc6,x86-64) => /lib/libc.so.6
	libanl.so.1 (libc6,x86-64) => /lib/libanl.so.1
	libBrokenLocale.so.1 (libc6,x86-64) => /lib/libBrokenLocale.so.1
	ld-linux-x86-64.so.2 (libc6,x86-64) => /lib/ld-linux-x86-64.so.2
Cache generated by: ldconfig (GNU libc) stable release version 2.36

@dlorenc I'm reading this now. And my ID is @tisonkun, you have some typos above XD.

@dlorenc I'm reading this now. And my ID is @tisonkun, you have some typos above XD.

Fixed! And now that you're here autocomplete works :)

➜  hawkeye git:(main) docker login ghcr.io
Username: tisonkun
Password:
Login Succeeded
➜  hawkeye git:(main) docker pull ghcr.io/dlorenc/images/java-static
Using default tag: latest
Error response from daemon: denied

I'm trying to build my native image docker image with it, but it seems fail to pull. Sorry I'm not quite familiar with ghcr.io. Do you have some ideas here?

My mistake - I forgot I had to make every new image public when I push to ghcr.io - can you try again?

@dlorenc Cool! It works.
You can try my demo by cloning https://github.com/korandoru/hawkeye and run:
docker build . -t hawkeye-native -f Dockerfile.native
docker run -it --rm hawkeye-native

The software is rather simple but still a completed native image and it shows that your image meets the basic requirement to run native images produced by GraalVM 22.3.0.

... and the final image size is 22.9MB. Awesome!

Awesome! I'll get a PR pushed up here with it :)

@dlorenc Thanks. Please notify me if you release the image at some stable source. To replace ubuntu:jammy I guess I need some reliable source instead of an one-time image - I cannot maintain it, yes :D

@dlorenc better advertise it in graalvm user group / community to get more feedbacks maybe.

The stable location for this image is now at cgr.dev/chainguard/graalvm-native-image-base!


Optimization: Keep Connection to Remove DCUtR peers alive

[ ] track identify messages and look for changes in advertised relay addresses

I assume you mean "remote" DCUtR peers? Just to help my understanding, could you briefly elaborate why this is needed?

The optimization is meant for the honey pot, not for the hole punching clients.
Keeping the connections to the (potentially behind a firewall) remote peer alive, as the honey pot, also keeps the identify protocol stream to the remote alive. Keeping the identify protocol stream alive implies receiving identify updates, in particular new relayed addresses, from the remote peer.
Does that make sense? Happy to expand on the reasoning.


Backup data stored on viame.kitware.com on external hard disk manually when script run

Include both files stored in girder and copied direct to viame hard drive

Backups of user-uploaded data now run nightly via duplicati.


FrontEnd: make a button to copy code snippet

The site is in React and it is available on here.
It shows the code snippets on the right-hand side. It is a reusable component.
In this component, Add a button to copy the given code snippet just like many amazing UI libraries have.
If you have any doubts, comment below!

I feel we can use the package [(https://www.npmjs.com/package/react-copy-to-clipboard) ]here.
Am I correct ,pls guide me ,I am new to this.

Working on it without a package as this is a fairly simple task

@Apoorva-13 the component is already there and you have to merge it there so as long as the website works it's okay to use any package. you simply have to add a button at the top corner of the div which copies the code.

@altany and @Apoorva-13, check your site locally and then submit PR. I will merge them.
develop the copyToClipboard component reusable and pluggable to keep the code modular.

@drex44 I just sent a PR. Luckily I created as a separate pluggable plugin, as you suggested, and also added it to your page. Hope it is what you had in mind and you approve :)

@Apoorva-13, are you still working on this?

No I am not working working on this.

okay @Apoorva-13, then I will close this issue!


Add node 7.10.1 Please

pkg 4.2.4
Linux 16.03 x64
ran: pkg -t node7.10.1-linux-x64
`

<EMAIL_ADDRESS>Fetching base Node.js binaries to PKG_CACHE_PATH
fetched-v7.10.1-linux-x64    [                    ] 0%
Error! 404 Not Found
https://github.com/zeit/pkg-fetch/releases/download/v2.3/uploaded-v2.3-node-v7.10.1-linux-x64
Asset not found by direct link:
{"tag":"v2.3","name":"uploaded-v2.3-node-v7.10.1-linux-x64"}
Not found in GitHub releases:
{"tag":"v2.3","name":"uploaded-v2.3-node-v7.10.1-linux-x64"}
Building base binary from source:
built-v7.10.1-linux-x64
Cloning Node.js repository from GitHub...
git                          [====================] 100%
Checking out v7.10.1
Applying patches
Compiling Node.js from sources...
make                         [                    ] `

The make process does not seem to do anywork.. stays at 0%, have let sit run for some time.
Just wanted to add these patches to my executable.
https://nodejs.org/en/blog/release/v7.10.1/
Thank you for your time and diligence.

ARMv6 32-bit Binary: https://nodejs.org/dist/v7.10.1/node-v7.10.1-linux-armv6l.tar.xz
ARMv7 32-bit Binary: https://nodejs.org/dist/v7.10.1/node-v7.10.1-linux-armv7l.tar.xz
ARMv8 64-bit Binary: https://nodejs.org/dist/v7.10.1/node-v7.10.1-linux-arm64.tar.xz
Is there a path to allow me add these to the .pkg-cache/ ??
Thanks again!

My bad..
make did work.. IT was just on -quite i saw in ps -ef
It finished and looks to have installed correctly.
Closing.


Add better stacking of configuration

This PR provides proper support for layering configuration values and should make it easier to build out some important features such as watching and rollbacks in the future.
I've created a new Configuration struct and module to provide all of the layering functionality. I think this is a terrible name so we should probably change it. The main role of Configuration is to hold onto a list of values that have been resolved from whatever provider. It can collapse these values into a list of keys which can then be easily loaded into an ets table. It can also maintain previous versions. Currently the versions are unused and only there for demonstration purposes. I suspect that realistically we'll want to use some sort of circular buffer or other bounded data structure to hold versions.
I've also created a Plan module for building up these configuration plans. Personally I find this to be a very shallow module but for some reason I felt compelled to keep those things separate for now. If anyone has better ideas here I'm happy to hear them.
Let me know what you think. @jeffweiss I know that you had some thoughts on this as well so let me know if this jives well with what you've been playing around with.

This is superseded by #33.


Host models on ivadomed OSF?

Currently, ivadomed models are hosted under SCT's OSF, which does not make sense for people who will not be using SCT. I suggest we host the models under ivadomed's OSF repository.

After internal discussions, we decided to host "production" ivadomed models on Github, following the approach used in SCT: https://github.com/neuropoly/spinalcordtoolbox/pull/2733
Pros:

Github's access worldwide is less restricted than OSF
We can version-track each model, assign a specific tag to it.


fix #111: real estates: basic add, update & delete functions

fix #111

@all-contributors please add @nmathey for code


Allow negative numbers as default arguments to work correctly

Fixes #562
The problem was that when constructing the R wrapper to the compiled function, the insertion of default values failed for negative values.  This occurred because of use of as.character where deparse is always safer.  as.character( quote(-1)) return a vector of "-" and "1" because "-" is handled as a unary negative operator, so the "-1" is parsed as a function call instead of a number.  This meant the code constructed was of the form function(x = -) {...}.   deparse(quote(-1)) returns "-1" as desired and allows construction of function(x = -1) {...}.

The test that failed on travis was fine on my machine.  To check if travis simply crashed, I restarted the tests.


⚠ Samsung has degraded performance

In 8a5a03f, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2500 ms

Resolved: Samsung performance has improved in f8c7240.


Jieba

Using jieba to extact input tags,before best_match.
For example, suppose there are 10000+ corpus in mongo, while user_input is "where is the apple?"
Before:
The best_match logic get all the corpus in mongo (distinct by in_reponse.text) , the size is about 10000 .Then we compare all this corpus with input. This takes a lot time.
After:
This pull extracts tags of the input. for this example,tags are maybe "apple". Then we add a addtional search option to mongo, using regex ,to select the corpus only related with "apple". the size is more less than 10000 (maybe only 100). Then we compare this 100 corpus with input. This is more fast than before.
Notice:
1.JiaBa required.
2.Best perfomance with Chinese, compatible with english.
3.The first time we load jieba takes about 2s,for each bot.

Find a similar pull with NLTK:
https://github.com/gunthercox/ChatterBot/pull/945
However, mongodb only support full text search with Chinese language in the enterprise edition.


Allow Bastion update of finalizer

How to categorize this PR?

/area quality
/kind bug
What this PR does / why we need it:
This PR allows the Bastion finalizer to be updated, even if the corresponding Shoot is in deletion. Otherwise it is not possible to get rid of the Bastion.
This PR also forbids to change the referenced Shoot on the bastion
Which issue(s) this PR fixes:
Fixes #5344
Special notes for your reviewer:
Release note:

The finalizer on the `Bastion` resource in the garden cluster can now be updated even if the referenced `Shoot` is in deletion

/assign


How to handle energy dependent deadtime/livetime?

The deadtime (or livetime) for CTA is in principle energy dependent (as different telescopes and cameras will have different deadtimes). The question is how to handle that. Fold the deadtime into the IRF? Of having explicitly an energy dependent deadtime? This is even worse if there are different camera types on the same site, and events seen with one camera type will have a deadtime different to events seen by other camera types.

Does the per-telescope deadtime really matter?

No, at DL3 per-telescope quantities don't matter.
The data model is EVENTS and IRFs for sub-arrays.
Everything per-telescope should happen at lower levels as part of the DL3 production.
One thing that is missing in the current DL3 spec is "event types", sub-partitions of the events by quality. These could be split by trigger pattern or other things. Each has their own effective area and deadtime fraction.

I think nothing is done in Prod2 or Prod3 with the deadtime, Gernot may confirm.
As I understand so far from discussions with Karl and Gernot, deadtime depends on trigger pattern (which cameras triggered, hence on energy) and time (for example rising moon). On VERITAS the deadtime is computed on a minute time scale.
If for a given sub-array a camera is dead while an event occurs, the multiplicity will be reduced. Hence without deadtime correction, the trigger pattern is different between MC and real world. Another related question is whether telescopes that have not triggered are actually used in the reconstruction (in the sens that it is verified that all telescopes that should have seen a shower have actually seen it). Then it becomes important to know when exactly a camera has been dead.
I agree that this is all low-level stuff, and should not be exposed at DL3 level. But I think the way how deadtime is computed will actually impact the way how it's stored in DL3.

No deadtime was taken into account in prod2/prod3 (we did some test on the impact of a 10% dead times for the NectarCam MSTs, do make su that this doesn’t have impact on the array layout selection)

On 20 Jun 2016, at 15:30, Jürgen Knödlseder<EMAIL_ADDRESS>wrote:
I think nothing is done in Prod2 or Prod3 with the deadtime, Gernot may confirm.
As I understand so far from discussions with Karl and Gernot, deadtime depends on trigger pattern (which cameras triggered, hence on energy) and time (for example rising moon). On VERITAS the deadtime is computed on a minute time scale.
If for a given sub-array a camera is dead while an event occurs, the multiplicity will be reduced. Hence without deadtime correction, the trigger pattern is different between MC and real world. Another related question is whether telescopes that have not triggered are actually used in the reconstruction (in the sens that it is verified that all telescopes that should have seen a shower have actually seen it). Then it becomes important to know when exactly a camera has been dead.
I agree that this is all low-level stuff, and should not be exposed at DL3 level. But I think the way how deadtime is computed will actually impact the way how it's stored in DL3.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Dr Gernot Maier
Deutsches Elektronensynchrotron (DESY)
Platanenallee 6, D-15738 Zeuthen, Germany

Today @moralejo and I discussed about this issue, and the only plausible solution we came up with was to include the dead time per telescope directly in the MC simulation. Then the deadtime/livetime time is not needed anymore, as the effective areas already contain the effect of the different dead times per telescope type. Then we should make sure that within the GTIs, data periods with the nominal dead time per telescope type are used.
@kosack @GernotMaier Do you agree with this approach? This would make science tools "blind" to the deadtime, which is probably the objective.

Maybe I misunderstood the idea, but including the dead time in the MC is something I don't feel is necessary:

what we care about is the effective/average deadtime for the whole system (or subarray).
this value depend both on the characteristics of the telescopes / cameras and environmental conditions (mainly NSB).
it will be a significant challenge to model the dead time of the system (and it also might depend on factors no included in the MC; e.g. buffer sizes in the readout). I don't see a necessity for this given that we will get precise dead time values from ACTL.

I really think this should be an item clearly separated from the MC.

For IBIS/ISGRI on INTEGRAL, the value of the dead time keyword is a header is an average of the 8 modules, but the set of keywords relating to ONTIME and EXPOSURE, are not consistent with the DEADC as one cannot reconstruct them using the others.

ONTIME is the sum of GTI (GTI calculated per module).
DEADC is an average, since there is one dead-time per MCE, the one used in calculating ONTIME. Therefore, one cannot get EXPOSURE = ONTIME*(1-DEADC).
EXPOSURE is ONTIME*MEANTIMEEFF.
MEANTIMEEFF is the percentage of average effective ISGRI time, taking into account the GTI and dead time per MCE. Therefore, it is not exactly equal to (1-DEADC).

Now, I gathered this information from the current PI, but nobody else knows this in detail, because the coders have come and gone many years ago. Would be nice if this information was in the fits files, and not scattered all over the place in different files produced at different stages of the pipeline. So, I think it should be included, in some way, into the final product that will be used by people, image, spectrum or time series file.

Let me explain the issue of "absorbing" the deadtime into the MC-calculated Aeff.  It is indeed dead time per telescope which matters, because there is no well-defined "system dead time", even if you make it energy dependent. In the very large SST array you can even have simultaneous events in distants parts of the array.
Consider this example: you may have an event triggering SSTs 30, 31 and 32, and, following shortly afterwards, another one that should have triggered e.g.  32, 33 and 34  - but 32 was busy with the previous event, and only 33 and 34 end up in the data stream. It is not that you lose the event, you have it, but with a smaller multiplicity.  In the least, you would need to make a toy MC to understand the impact of such single-telescope dead times in the "quality" of the recorded events. Since we make an MC anyway, why not to include this single-telescope dead time into it?
I understand that the dead times will be small enough to make the kind of event I mentioned above rather rare, and perhaps that makes this irrelevant for practical purposes (I really do not know). But implementing the telescope-wise dead time in the MC is probably so simple that there is no big reason not to do it.

Ok - didn't think of that. I have little feeling how big the effect is, would be nice to see a study on that. The effect is probably small, but if not, dead time would not only affect the acceptance, but also the energy and angular resolution. If this is the case, it would anyway need to be included in the MC.
If it is simply an energy dependent correction factor: why not keeping it separately? It would keep things just cleaner and more obvious (but your suggestion would also work).

If it is simply an energy dependent correction factor: why not keeping it separately? It would keep things just cleaner and more obvious (but your suggestion would also work).

What about divergent pointing? It would also depend on the FoV, and it would become an absolute nightmare to factorize it.
Including it in the MC is probably the safest and easiest solution to implement.

@jknodlseder @GernotMaier @cdeil @moralejo
Did we converge on this? With an energy/FoV dependent deadtime, I see no easy solution to incorporate it into the DL3 unless we directly include it into the MC (and therefore, already inside the IRFs).
If we all agree with this solution (for the specific case of CTA), should it be written somewhere?

What we have now (a single number DEADC in the EVENTS header) works for all existing instruments, no?
Yes, in the future CTA might have something more complex, but IMO we could just wait for such data to appear and then define a format for it.
@TarekHC - If you want to define something now, I think this would work best: talk to @GernotMaier and @kosack or others from CTA first (probably easiest in a telcon, not via Github issue, this is a too complex topic) to make sure it covers what you expect CTA will need, i.e. what kind of info CTA MC / pipe will produce in the future, and then come back with a proposal. Assigning this issue to you.

Fine with me.
For reference, here are the relevant requirements for CTA:
A-PERF-0810 System Deadtime
Description:
The fraction of the time that the system as a whole (i.e. elements of the system may be unavailable for longer periods) is unavailable for recording of events, due to inefficiency in data collection, transport and storage, during observations (with telescopes ontarget) must be <2 %.
A-PERF-2250 Telescope Deadtime
Description:
The fraction of the time that an individual telescope is unavailable for recording of events (due for example to inefficiency in data collection, transport and storage) during observations must be <8 %.

Just a note: there was some discussion that's related to this issue in #97.
I briefly re-read the discussion here, and to be honest I'm not sure if any change is needed to what we already have in the DL3 spec for CTA. DL3 is only concerned with array-level AEFF and BKG rate, per-telescope deadtime concerns are lower-level, there are no per-telescope parameters in DL3. (so some of the discussion above is relevant for CTA, but concerns what MC / pipeline should do to produce DL3, not something that should be stored in DL3 files).
Of course it's possible to split exposure = area x time in multiple ways, putting the deadtime fraction with area or time to get the effective exposure that is needed in DL3. But I don't see why any other way to do it would be better to what we already have now (which is to have exposure = area x livetime with livetime = (1 - DEADC) x obstime, where DEADC is a single number (not energy-dependent) per time interval. CTA (and other instruments) can have more info internally like energy-dependent deadtime fractions or whatever they like for debugging, but there's not point in creating a more complex DL3 spec and having to teach science tool codes about these things, no?

For the record: I've changed my mind on this.
As already mentioned in https://github.com/open-gamma-ray-astro/gamma-astro-data-formats/pull/97#issuecomment-428214663 my suggestion for IACT data would be to compute AEFF and BKG rate models at the DL3 level, i.e. as given to the science tools, so that they include all dead-time effects, and DEADC=1, i.e. LIVETIME=ONTIME.
This gets rid of the question / confusion when to use which time for exposure and background model count computation, there is only one time.
It also gets rid of the question how to store this in DL3, where currently the DEADC key is in the EVENTS header, but then the start / stop observation times are in a different FITS HDU, the GTI.
This is a simple and generic solution for the DL3 interface, and it doesn't limit options at all what to do at the MC and pipeline level.
Note that currently ctools and Gammapy are handling background rates differently (ctools simulates with rate per livetime, Gammapy analyses assuming rate per obstime), and because counts are so high in CTA, these effects are visible in the simulated DC-1 data (see https://github.com/gammapy/gammapy/issues/1842#issuecomment-427679687).
@kosack or anyone - any chance to get a decision on this for CTA DL3 or create a decision process so that this and other questions can be decided in CTA soon?


Support CUDA Graph for MoE models

Description
Different from non-MoE models like llama2, MoE models have dynamic shaped activations in FFN layers, so one cudagraph can only capture a part of one transformer layer, instead of covering the whole layer. We call this a "breaking-layer" cudagraph mode. This PR adds breaking-layer cudagraph supports for MoE models on the TE side, and fixes several related bugs in TE.
Fixes # (issue)
Type of change

[ ] Documentation change (change only to the documentation, either a fix or a new content)
[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Infra/Build change
[ ] Code refractor

Changes
Please list the changes introduced in this PR:

Add is_initialized() method in CudaRNGStatesTracker to align with what is already done in MCore.
Fix wrong per_callable_module_params order bug in _make_graphed_callables when _order is given.
Fix warmup argument mismatch bug in _make_graphed_callables when _order is given.
Fix fp8 accuracy issue by adding fp8_group argument to make_graphed_callables() and modifing is_first_microbatch, skip_fp8_weight_update and fp8_meta code.
Support MoE models cudagraph by filtering graphed TE modules and model weights during warmup.

Checklist:

[x] I have read and followed the contributing guidelines
[x] The functionality is complete
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

/te-ci pytorch

/te-ci pytorch

Have you tested that it works with Mcore?
Yes, we also made some changes in Mcore to enable MoE cudagraph. You can refer to issue 193 in our Megatron-LM repo.

/te-ci pytorch

/te-ci pytorch

/te-ci pytorch

/te-ci pytorch

/te-ci pytorch

Hi @timmoon10 , do you have more suggestions on this PR?

/te-ci pytorch L1

Merge this PR since pipeline 20710146 passed and Tim approved.


Ionic2 - Failed to load resource: The requested URL was not found on this server on iOS

Note: for support questions, please use one of these channels:
https://forum.ionicframework.com/
http://ionicworldwide.herokuapp.com/
Short description of the problem:
Unable to run the app correctly in iOS if  ionic build --prod is run.
In the console, the error msg is:
Failed to load resource: The requested URL was not found on this server. file:///MY_PROJECT_ABSOLUTE_PATH/www/build/main.js.map Failed to load resource: The requested URL was not found on this server.
All labels translated from the ngx-translate are missing, but the other function seems fine.
However, no one can deliver an app without labels nor super slow.
What behavior are you expecting?
Not breaking in iOS.
Steps to reproduce:

ionic build --prod
run in iOS

insert any relevant code between the above and below backticks

part of the package.json
  "config": {
    "ionic_rollup": "./config/rollup.config.js",
    "ionic_copy": "./config/copy.config.js",
    "ionic_generate_source_map": true,
    "ionic_bundler": "rollup"
  },

part of the rollup.config.js
  entry: process.env.IONIC_APP_ENTRY_POINT,
  sourceMap: process.env.IONIC_GENERATE_SOURCE_MAP ? true : false,
  format: 'iife',
  dest: process.env.IONIC_OUTPUT_JS_FILE_NAME,

Which @ionic/app-scripts version are you using?
Cordova CLI: 6.5.0
Ionic Framework Version: 3.0.1
Ionic CLI Version: 2.2.1
Ionic App Lib Version: 2.2.0
Ionic App Scripts Version: 1.3.7
ios-deploy version: Not installed
ios-sim version: Not installed
OS: Linux 3.19
Node Version: v6.7.0
Xcode version: Not installed
Other information: (e.g. stacktraces, related issues, suggestions how to fix, stackoverflow links, forum links, etc)

We noticed someone has filed a similar issue #805 while he is using webpack + osx, but we are using rollup + Linux and still found the same issue.
Our app works fine in ionic 2. We experienced this problem after upgrading to ionic 3.
We have to force rollup to generate source map or the script won't compile(in ionic 2+3).
We only experienced this problem in iOS devices. In android devices, even the source map path is written as absolute path in the main.js, our app works fine.
We tried to manually replace the absolute path to relative(/MY_PROJECT_ABSOLUTE_PATH/www/build/main.js.map -> /build/main.js.map). The error message changes to "file:///build/main.js.map Failed to load resource: The requested URL was not found on this server." For some reason, iOS insisted to use absolute path and not willing to provide the app storage prefix. So it gives us no choice but file an issue in here

Actually, we found that our developers is missing a slash during path replacement. Our app run fine in iOS AFTER the path replacement. Sorry for your time.


Add option to bypass error - Multiple CORS header 'Access-Control-Allow-Origin' not allowed

Implement an option, disabled by default, to fix the following error on the client side:
Reason: Multiple CORS header 'Access-Control-Allow-Origin' not allowed

Reason
Reason: Multiple CORS header ‘Access-Control-Allow-Origin’ not allowed
What went wrong?
More than one Access-Control-Allow-Origin header was sent by the server. This isn't allowed.
If you have access to the server you can change your implementation to echo back an origin in the Access-Control-Allow-Origin header. You cannot send back a list of origins, because browsers only accept a value that is either a single origin or null

Possible candidates for further options here: CORS errors

test

zzzzz

zzzz

zzz


Make bst-here more flexible

See original issue on GitLab
In GitLab by [Gitlab user @tlater] on Apr 24, 2018, 11:03
Having used bst-here for a little while now I find that while it works quite well, it lacks some flexibility in terms of the system running in the container.
It would be nice if:

We could support changing the image used for the container. We should still only support buildstream/buildstream-fedora upstream, but users may write plugins that require different host tools. Such users might need to use a different image to use bst-here, and currently need to hand-edit the file.

I propose adding both a -i CLI flag as well as a BST_HERE_DOCKER_IMAGE environment variable, so that the user can set a default, but there's still a way to change it temporarily.

We could support adding additional volumes. This might be nice to, for example, include keys or test environments into the container.

This would probably work best if we mimic the docker -p flag, since we are already trying to be somewhat consistent with the docker CLI.

In GitLab by [Gitlab user @tlater] on Apr 24, 2018, 11:03
changed the description

In GitLab by [Gitlab user @tlater] on Apr 24, 2018, 11:04
changed the description

In GitLab by [Gitlab user @tlater] on Apr 24, 2018, 11:04
I discussed these changes with [Gitlab user @cs]-shadow, probably worth pinging here :)

In GitLab by [Gitlab user @tlater] on Apr 24, 2018, 11:05
changed the description

In GitLab by [Gitlab user @cs-shadow] on Apr 26, 2018, 23:41
created branch 378-make-bst-here-more-flexible

In GitLab by [Gitlab user @cs-shadow] on Apr 26, 2018, 23:41
mentioned in merge request !439

In GitLab by [Gitlab user @cs-shadow] on Apr 26, 2018, 23:43

This would probably work best if we mimic the docker -p flag, since we are already trying to be somewhat consistent with the docker CLI.

[Gitlab user @tlater], to clarify, did you mean to say -v (--volume) instead of -p?

In GitLab by [Gitlab user @cs-shadow] on Apr 27, 2018, 00:37
mentioned in commit b5d2086eb05e0605fc6a93b658db5940fe1407aa

In GitLab by [Gitlab user @cs-shadow] on Apr 27, 2018, 00:37
mentioned in commit d1c9cce8802d166954efc9b5256568aeb6225370

In GitLab by [Gitlab user @cs-shadow] on Apr 27, 2018, 12:35
mentioned in commit 25719d55a94f27981ea060c73d7aa8313a54329c

In GitLab by [Gitlab user @cs-shadow] on May 7, 2018, 12:21
mentioned in commit 6ffca067b22c625bb2a25cf0494b7ec022b2fa99

In GitLab by [Gitlab user @cs-shadow] on May 7, 2018, 12:21
mentioned in commit 491760395eb9cb1e27fbe62ec49d98af7cca0d61

In GitLab by [Gitlab user @tristanvb] on May 7, 2018, 12:39
closed via merge request !439


Patrol command doesn't work

Guards ignore it and just continue roaming around as they normally do, regardless of whether the player has HOTV or not.

fixed in 1.1.0


feat: add english and spanish automatic translation

base on the user settings defined in the browser the content will be return in english or spanish, with english as fallback for any other language

:tada: This PR is included in version 2.3.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


⚠ Everything Hub has degraded performance

In 3987ab7, Everything Hub (https://hub.everythingbagel.me) experienced degraded performance:

HTTP code: 200
Response time: 3500 ms

Resolved: Everything Hub performance has improved in 20496e8 after 5 minutes.


Question about LPIPS percentage point metric

Dear author,
I have read "Stable View Synthesis" and wonders what is LPIP pertancage point metric.
In general, LPIPS measures the perceptual distance in Euclidean and even your another paper "Free View Synthesis" report LPIPS, not LPIPS percentage point.
Can you explain how LPIPS percentage point is calculated ?
And if LPIPS percentage point is relative to the best prior state of the art, why lower LPIPS percentage point is better than higher ?
Thank you for your great work!
Best regards,
YJHong.

LPIPS% = 100 * LPIPS.
As you have written, LPIPS is a Euclidean distance in feature space, hence, lower is better.

@griegler
Thank you!
Close the issue.


New ztest fx cannot specify different setup/teardown functions for each test function within the same test suite.

Is your feature request related to a problem? Please describe.
In code like below, the ztest_1cpu_user_unit_test() and ztest_user_unit_test() macros register test functions with different setup/teardown functions.
https://github.com/zephyrproject-rtos/zephyr/blob/ff54de3e1697e50f05cb8354f5a4ddb548107b31/tests/kernel/mutex/mutex_api/src/test_mutex_apis.c#L397-L403
https://github.com/zephyrproject-rtos/zephyr/blob/f2ae4b67b2ab6efeda116bd362cd078f2b1e5793/subsys/testsuite/ztest/include/zephyr/ztest_test_deprecated.h#L242-L243
https://github.com/zephyrproject-rtos/zephyr/blob/f2ae4b67b2ab6efeda116bd362cd078f2b1e5793/subsys/testsuite/ztest/include/zephyr/ztest_test_deprecated.h#L215-L216
This is because the old ztest fx records setup/teardown functions at the test function level.
https://github.com/zephyrproject-rtos/zephyr/blob/f2ae4b67b2ab6efeda116bd362cd078f2b1e5793/subsys/testsuite/ztest/include/zephyr/ztest_test_deprecated.h#L24-L30
But with the new ztest fx, setup/teardown can only be recorded at the test suite level as the before/after functions. Like below:
https://github.com/zephyrproject-rtos/zephyr/blob/f2ae4b67b2ab6efeda116bd362cd078f2b1e5793/subsys/testsuite/ztest/include/zephyr/ztest_test_new.h#L114

This will cause inflexibility for test suite arrangement.
We have to separate test functions based on their setup/teardown functions even they belong to the same test suite logically.
Describe the solution you'd like
Maybe we should be able to specify setup/teardown functions for each individual test function.

I am not sure if this should be labeled as a bug or feature request. Please help remove the inappropriate one. Thx.

May be fixed by: https://github.com/zephyrproject-rtos/zephyr/pull/47660

@aaronemassey @nashif
This was a very poor design by the original ztest and I would very much not like to see us going back in that direction. Prior to the big refactor of Ztest test suites actually didn't mean anything. All the before/after logic was bound to the test and the suite basically just provided some kind of name scoping.
Looking at other test frameworks as a model, the point of suites is to abstract away common code. If you need custom logic before/after your individual test, then that's not common code.
Basically, if you find yourself doing the same thing in multiple tests, use a suite. If you need to do things differently for just one test, just include that in the start of the test.
I'm leaving this issue open for visibility, but would like to see this marked as "working as intended".

I think test suite should just be a logic container of all the test functions targeting the same feature. That doesn't necessarily mean test functions mustn't do anything specific to themselves.
It is possible to create yet another test suite with 1cpu specific before/after functions. But in doing that we are merely grouping test suites for the sake of the implementation of the framework, rather than by the feature we are testing.

If we take this as a reference: Test fixture
Test fixtures can be set up three different ways: in-line, delegate, and implicit.

In-line setup creates the test fixture in the same method as the rest of the test. While in-line setup is the simplest test fixture to create, it leads to duplication when multiple tests require the same initial data.
Delegate setup places the test fixture in a separate standalone helper method that is accessed by multiple test methods.
Implicit setup places the test fixture in a setup method which is used to set up multiple test methods. This differs from delegate setup in that the overall setup of multiple tests is in a single setup method where the test fixture gets created rather than each test method having its own setup procedures and linking to an external test fixture.

This issue is asking for  the delegate solution.
The 1cpu thing could be seen as a fixture that only some tests need, right now we are using this per testsuite, which is in my opinion not that great and forces the creation of additional suite just for 1cpu. Obviously we could use the in-line solution and in my early tests that is exactly what I did, but now we have many tests using the before/after of  suites to accomplish this.
@yperess  which suites did you look at for reference?

pytest has the above per testcase, Robot Framework as well

I assume the pytest flow you mean is:
import pytest

@pytest.fixture
def my_cleanup_fixture():
    # Startup code
    ...
    yield
    # Cleanup code
    ...

@pytest.mark.usefixtures('my_cleanup_fixture')
def test_with_special_cleanup():
    pass

Where the fixture is added to the test_with_special_cleanup() using the annotation pytest.mark.usefixtures? If so, I'm still not seeing where this is a missing feature. For example, while teardown is a bit harder, if you really didn't want to split things off into a separate suite, you could do the same via:
struct my_suite_fixture {
  bool is_1cpu;
};

static void *my_suite_setup(void) {
  static struct my_suite_fixture;

  my_suite_fixture.is_1cpu = false;
  return &my_suite_fixture;
}

static void my_suite_before(void *fixture) {
  ((struct my_suite_fixture *)fixture)->is_1cpu = false;
}

static void my_suite_after(void *f) {
  struct my_suite_fixture *fixture = f;

  if (fixture->is_1cpu) {
    z_test_1cpu_stop();
  }
}

ZTEST_SUITE(my_suite, NULL, my_suite_setup, my_suite_before, my_suite_after, NULL);

ZTEST_F(my_suite, test_x) {
  z_test_1cpu_start(); //< This is the same as adding function specific before
  fixture->is_1cpu = true;
}

As per testing frameworks:
JUnit, TestNG, NUnit, Catch2, gTest, boost, C# tests (which I forget the name of the framework), there are more but I think it's enough.
Ones that do support it:

XCTest - Apple's testing framework for swift, to achieve this they use a DSL to extend the test function such that blocks can be created inside the test function:

func testMethod() throws {
  addTeardownBlock {}
}

I'm sure there are others, but see below, they are all for higher level functions that support annotation and/or DSLs

Overall, the languages that do support these features are highly flexible (considerably more so than C or even C++). Adding yet another entry point to the ZTEST* macros makes the API messy and hard to read, Unlike the examples above that do support these features, we do not have the benefit of annotations or DSLs. Basically, the cost tradeoff from what I'm looking at is:

It makes the API more complex
It doesn't add a feature that can't already be done

I've been toying with an idea that I think might solve the 1cpu issue, but it isn't as flexible as per test before/after.
We could simply provide bool is_1cpu() and check it in the ztest logic. If it returns true, we can call z_test_1cpu_stop() for the test.

@yperess  I also do not like the idea of ZTEST_1CPU although I suggested it on discord the other day, this does not scale if we have something else similar to 1CPU and adding those to the API just does not make sense, I agree. But we need to have a way to deal with per testcase before/after or something else that is not 1cpu specific and more of a general solution.
We should not rush into solutions, let think about it a bit more and wait for some more results from the migration to the new API, we can then address all issues at once in a more orgnized fashion rather than hot fixes and worarounds.

@nashif I had another thought just now, we could actually do something like XCtest, ztest could support a single linked list of cleanup functions that we can register when the test is running, in a very rough example:
ZTEST (s, test_x) {
  ztest_add_cleanup_function(test_x_after);
}

I would want to pass the fixture when available, but I haven't quite thought of how.

@yperess
Registering a linked list of cleanup functions with some new API like ztest_add_cleanup_function() or using the static void my_suite_before(void *fixture) in the example above seem to make things unnecessarily complicated/twisted.
Adding 3 more ZTEST_*_WITH_BEFORE_AFTER macros won't make the API more messy IMO. Rather it offers a flexible and intuitive way for test writers to fine tune each test function. And we shouldn't limit ourselves to 1cpu too much. I think https://github.com/zephyrproject-rtos/zephyr/pull/47660 is generic enough for other similar scenarios.
BTW, I briefly checked the frameworks you mentioned. TestNG seems to support before/after functions at different levels.
So maybe it's not that a bad idea. :)

Afaik, TestNG's before/after will run on every function in the class/suite. https://howtodoinjava.com/testng/testng-before-and-after-annotations/#1-testng-before-and-after-annotations explains it pretty well.

Yeah... it's 4 more... miscounted... 😵
ZTEST_WITH_BEFORE_AFTER
ZTEST_USER_WITH_BEFORE_AFTER
ZTEST_F_WITH_BEFORE_AFTER
ZTEST_USER_F_WITH_BEFORE_AFTER

I agree that we should be cautious about API design.
Let's hear more from the community and see how to balance between intuitiveness and purity.

Current before/after functions may run on different threads, which may cause potential issues. See: https://github.com/zephyrproject-rtos/zephyr/issues/48147


Check the README.md updates for the direction I'm heading in.
This may or may not be suitable for merging as-is.
Feedback as to the direction is welcome though (and certainly the way I discover firmware files needs to be tested a bit)

This change is

Looks like in common/ OpTestConstants.py lpar name still not changed to host.

Overall direction looks good to me.  I should have known better with the LPAR naming :)  If we can get @pridhiviraj or @saqibkh to verify no regressions (and you resolve the merge conflict) then we should be good to merge.

Later today I'll do another iteration, rebasing on master and fixing up comments made here. I'll also likely flesh out the reporting output so that everything goes into out/

Okay... so here's a new (rebased) go at it. I go further in this set, converting everything to output and consume TAP (Test Anything Protocol). This gets us a machine parseable output and human readable one.
Now, there's all the output you need to look at test failures in out/

Executed the tests with these changes, all are working fine. some enhancements are needed from inputs point of view(hpmimage and ffdc dir) to make run script usable for all the bvt's. As of now everything looks good.

Reviewed 33 of 46 files at r1, 9 of 12 files at r2, 3 of 4 files at r3, 8 of 8 files at r4.
Review status: all files reviewed at latest revision, 2 unresolved discussions.

Comments from Reviewable

Hi Stewart,
You patch is causing the bvt runs to break. I'm not sure what the exact cause it but you can find the logs to recreate here: http://gfwa122.aus.stglabs.ibm.com:8082/job/FTC/job/OpenPower/job/Miscellaneous/job/Github_Patch_Test/30/console.
Let me know if you need me to copy this to some other location/

saqibkh<EMAIL_ADDRESS>writes:

You patch is causing the bvt runs to break. I'm not sure what the exact cause it but you can find the logs to recreate here: http://gfwa122.aus.stglabs.ibm.com:8082/job/FTC/job/OpenPower/job/Miscellaneous/job/Github_Patch_Test/30/console.
Let me know if you need me to copy this to some other location/

looks like --lparip is still being passed in, sohuld be --hostip etc.
I'd like to remove run-op-bvt/run-op-it too - and instead just launch
everything using the top level ./run script.
--
Stewart Smith
OPAL Architect, IBM.


Broker datapoints interface to retrieve metadata should include data type unit

The data unit types of VSS  should be exposed via GetMetadata as well.
Eg when subscribing to "Vehicle.Speed", the value is in "km/h" as specified. Since the spec can be overriden, it's hard for generic tools ("VSS Browser") to know the specific data type unit, unless it is exposed via the metadata interface.

As this seems to concern databroker it should possibly be filed at https://github.com/eclipse/kuksa.val/issues instead.
@argerus - could you possibly give quick feedback. Is it Databroker that does not send unit to clients, or is clients that does not show it. No change in proto required I assume?
Test Client> getMetaData Vehicle.Speed
{
    "path": "Vehicle.Speed",
    "metadata": {
        "data_type": "FLOAT",
        "entry_type": "SENSOR",
        "description": "Vehicle speed."
    }
}

sdv.databroker.v1 > metadata Vehicle.Speed
[metadata]  OK
Path          Entry type Data type
Vehicle.Speed Sensor     Float

@mikehaller @argerus - do we consider this one still relevant? If so I could "migrate" it to kuksa.val, or the new kuksa-databroker repo when migration is finished.

Still relevant imo.


Image for basic island land is not loading correctly

https://deckmaster.fugi.io/cards/17/67017.jpg
Here's a screenshot from a streamer
https://i.imgur.com/XtpuZHC.jpg

Should be fixed now, thanks for the report!


oadm depricated

The command oadm ... will be deprecated in future versions of openshift.
The supported command is now oc adm ...

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d6287c41-d51f-e744-4bb2-5a2f8dd818e7
Version Independent ID: 5df00963-19bd-e6f3-0f0e-74a483d207a5
Content: OpenShift on Azure post-deployment tasks
Content Source: articles/virtual-machines/linux/openshift-post-deployment.md
Service: virtual-machines-linux
GitHub Login: @haroldw
Microsoft Alias: haroldw

Thanks for the feedback! I have assigned the issue to the content author to investigate further and update the document as appropriate.

@git001 Thank you for the feedback! The author has made multiple updates to the doc since you provided your feedback. I think the issue has been addressed, so I am going to close this issue now.
Feel free to reopen this or open another issue if you still have concerns. Thanks again!
#please-close


update function's error check to match get.js pattern

Looks like I actually janked this one up. Will create new PR with correct code -
if (!result.Items.length === 0) {
throw new Error("No items found.");
}


Add Link Index to Person

Logic

Add link index to add person command

Codecov Report

Merging #97 (d57c0ea) into master (ecca12a) will decrease coverage by 0.22%.
The diff coverage is 53.33%.

@@             Coverage Diff              @@
##             master      #97      +/-   ##
============================================
- Coverage     54.85%   54.62%   -0.23%
+ Complexity      459      457       -2
============================================
  Files            92       92
  Lines          1907     1924      +17
  Branches        222      225       +3
============================================
+ Hits           1046     1051       +5
- Misses          797      809      +12
  Partials         64       64

Impacted Files
Coverage Δ

...ain/java/seedu/address/logic/parser/CliSyntax.java
92.30% <ø> (ø)

...in/java/seedu/address/logic/parser/ParserUtil.java
47.82% <0.00%> (-6.15%)
:arrow_down:

.../java/seedu/address/model/util/SampleDataUtil.java
9.67% <0.00%> (ø)

...seedu/address/logic/commands/AddPersonCommand.java
81.39% <66.66%> (-0.19%)
:arrow_down:

...u/address/logic/parser/AddPersonCommandParser.java
87.50% <66.66%> (-12.50%)
:arrow_down:

...eedu/address/logic/commands/EditPersonCommand.java
95.89% <75.00%> (-1.22%)
:arrow_down:

.../java/seedu/address/storage/JsonAdaptedPerson.java
87.50% <100.00%> (ø)

...rc/main/java/seedu/address/model/ModelManager.java
79.10% <0.00%> (-1.50%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


smtp to grpc gateway

Issue Template
Title: smtp to grpc gateway
Description:
I'm interested in implementing an smtp-to-grpc gateway as an envoy plugin. The plugin would transcode rfc5321 to a grpc/protobuf intra-cluster representation "smtp.proto". This would open up access to any language with grpc support; you can receive smtp by writing a grpc handler. I'm thinking that this approach could be useful in situations where the business logic you want to implement is simple but the wire protocol is non-trivial and there aren't modular/resuable implementations for many languages.
SMTP does an in-band TLS upgrade via the STARTTLS command https://tools.ietf.org/html/rfc3207 and this is probably a hard requirement for most users. AIUI juggling the transport on the fly like this is not possible in envoy today so adding support for that may be a heavier lift than smtp itself. I don't know off the top of my head how many other protocols work that way.

cc. @lizan

This should be doable within existing extension mechanism, with SMTP L4 filter + grpc bridge, it will be more generically reusable to split them to two.

SMTP does an in-band TLS upgrade via the STARTTLS command https://tools.ietf.org/html/rfc3207 and this is probably a hard requirement for most users. AIUI juggling the transport on the fly like this is not possible in envoy today so adding support for that may be a heavier lift than smtp itself. I don't know off the top of my head how many other protocols work that way.

Not sure how exactly STARTTLS works, but if you're able to handle it within listener filter, then the transport socket can be kicked in after that.

In smtp, the client says EHLO, the server returns a list of capabilities and then the client says STARTTLS. But there is still a long tail of clients that don't do starttls so it needs to work both ways.

@jsbucy right, that's the purpose of the listener filter, you can return list of capabilities there and see whether client says STARTTLS, then select right filter chain (TLS or not) based on that. It will be similar to tls_inspector but more complicated though.

Ok so we would have 2 filters: the "smtp starttls" listener filter that just responds to ehlo and starttls and then the "smtp transcode" network filter that does all of the grpc stuff?
I think it is technically valid for a client to send one smtp transaction in the clear and then say STARTTLS and send another one which would break this implementation. I can't imagine why the client would do that but I don't know for sure that noone does.

Ok so we would have 2 filters: the "smtp starttls" listener filter that just responds to ehlo and starttls and then the "smtp transcode" network filter that does all of the grpc stuff?

Sounds reasonable, "SMTP transcode" can be split to SMTP handling (like HTTP connection manager filter) and GRPC-SMTP L7 filter (like HTTP router filter).

I think it is technically valid for a client to send one smtp transaction in the clear and then say STARTTLS and send another one which would break this implementation. I can't imagine why the client would do that but I don't know for sure that noone does.

At a quick glance on RFC 3207 I don't think that's valid in term of spec? I doubt anyone has use case on that, so we can ignore that case unless someone will use it.

In this design, you'd configure the smtp capabilities list into the envoy plugin. The listener filter would

always return a 2xx success initial response (smtp "greeting") unless envoy itself had some kind of internal error
return the same set of capabilities to every client
always initiate tls in response to STARTTLS unless there was some kind of internal error

If it were possible to switch the transport socket on the fly, this could all be directed by the upstream/grpc backend which might want to customize behavior for different clients. Just for the sake of argument, what would it take to implement that in envoy? I've heard some discussion offline that it might not be a very extensive change?
I think we've identified a workable design here but just thought I'd ask. I don't know if anyone else could take advantage of this feature so I wouldn't put it on the critical path here unless it's straightforward and uncontroversial.

If it were possible to switch the transport socket on the fly, this could all be directed by the upstream/grpc backend which might want to customize behavior for different clients. Just for the sake of argument, what would it take to implement that in envoy? I've heard some discussion offline that it might not be a very extensive change?

Let's open a new issue for this topic, this is not impossible but it is heavy (not only implementation, and tests), basically you want to shutdown existing filter chain and use existing connection to instantiate another filter chain. I'm not sure how useful this is but worth discussion. @envoyproxy/maintainers WDYT?

So after staring at the TLS Inspector filter code I'm not sure that approach will work for smtp. The gist of how tls inspector works is it accumulates the TLS ClientHello message in the socket buffer (with MSG_PEEK) until we can "dry fire" SSL_do_handshake on it to parse out the SNI and ALPN.
In the case of smtp, the client will say STARTTLS immediately after EHLO say 90% of the time but I'm not sure how to make this work when the client doesn't say starttls. In that case, we need to send ehlo to the upstream (presumably with the saved argument the client sent us) and wait for the response before passing the next command. But the listener doesn't have access to the upstream connection? Judging based on the comments at the beginning of onRead:

This receive code is somewhat complicated, because it must be done as a MSG_PEEK because  there is no way for a listener-filter to pass payload data to the ConnectionImpl and filters that get created later.

It seems like another approach would be to write a wrapper for TransportSocket that is more "in the middle" of the bytestream that would watch the smtp session for STARTTLS or until the session changes out of line/command mode (auth/data/bdat) at which point it becomes a pass-through.

some progress on starttls on #9577


CIS changes: cis-1.5, profiles, schedule, alerts

Issues:
Schedule/Alert: https://github.com/rancher/rancher/issues/25326
Profiles: https://github.com/rancher/rancher/issues/25325
CIS 1.5 support: https://github.com/rancher/rancher/issues/24987


get_hotspot crashed after 4 hours

It ran for hours and then crashed...i removed the hotspot address from below.
helium_hotspot_exporter.py:req_get_json:86:DEBUG	fetching url: https://api.helium.io/v1/hotspots/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
helium_hotspot_exporter.py:req_get_json:88:DEBUG	fetch returned: <Response [503]>
Traceback (most recent call last):
  File "/opt/app/helium_hotspot_exporter.py", line 298, in <module>
    stats()
  File "<decorator-gen-1>", line 2, in stats
  File "/usr/local/lib/python3.9/site-packages/prometheus_client/context_managers.py", line 66, in wrapped
    return func(*args, **kwargs)
  File "/opt/app/helium_hotspot_exporter.py", line 286, in stats
    stats_for_hotspot(addr, hname)
  File "/opt/app/helium_hotspot_exporter.py", line 173, in stats_for_hotspot
    d = get_hotspot(addr)
  File "/opt/app/helium_hotspot_exporter.py", line 122, in get_hotspot
    return ret['data']
KeyError: 'data'

cool, figured I'd start seeing non-200s at some point. I'll fix.


📦 Export default and named package

This is to make JEST happy in downstream packages that use es6.

A test case would've been nice :-/


Add target nameplate options

Not ready yet.  Tests correctly on /showname 1 on accident.  When I switched to /showname 4, discovered need to adjust logic to account for Surname and Guild in Nameplate

This one is now ready to be reviewed with the most recent changes to account for two line Nameplates.  The buttons and permutations of the buttons all appear to work on my end


User Snippets multiple same name variables maintain multiple cursor positions on completion

VSCode Version: 1.5.2
OS Version: Win10

Steps to Reproduce:

Create user snippet with multiple variables of same name
Use the snippet
Press enter to complete while editing the variable that has multiple locations
VSCode maintains multiple cursor positions at all of the locations of the variable

After completing a user snippet there should only be one cursor position

@cpamp21 Can you please include here the actual snippet you're using?

Javascript.json:
"IIFE": {
	"prefix": "jsiife",
	"body": [
		"(function(${params:params}) {",
		"",
		"})(${params:params});"
	]
}

I see, this snippet does not contain an end cursor position, so at the end (when leaving snippet mode) the two cursors are still maintained and the only way to get rid of them is through Escape.
@cpamp21 Usually snippets contain $0 to indicate the end cursor position. Would using $0 work for you or would you necessarily expect that we remove extra cursors also for such snippets (that lack a $0)? i.e. I would change your snippet to (which makes sure the cursor is in a good spot at the end):
"IIFE": {
    "prefix": "jsiife",
    "body": [
        "(function(${params:params}) {",
        "\t$0",
        "})(${params:params});"
    ]

@alexandrudima Thank you. I would expect them to be removed but I think using $0 is preferred here.


once I start my repo, everware offers forking it

start my repo with everware (e.g. https://github.com/anaderi/openml-demo.git)
open control panel
it says You don't have a repository with the same name. Do you want to FORK it?

seems it is more appropriate to offer push right away.

This is related to #67. Can we close this one and collect all the discussion in #67?

this one looks like a bug and #67 is a generic discussion, no? I guess we might have plenty of bugs on github integration topic and putting all of them into a single issue might smell like a havoc, no?

Ah, sorry, I misread the above as "It would be nice if everware did this" not as "everware is doing this right now and it is doing the wrong thing". Thought we should collect the feature requests in one, but agree that we should have one issue per :beetle:.
Double checking: right now everware offers you forking a repo which you already own, which is a bit silly as it should just directly offer you to push to it.
The fix would be to add a step to detect if you already have a fork of the repository. There are two ways to list a user's repos: list all and list public. You can also list all the user's forks.
The home handler already has a method to check if a repo exists. I think it accesses the wrong URL though (can't find that one in the API docs). @ibab do you remember?
We need to take care to detect the case of the user starts organisation/repoA, is able to push to organisation/repoA but has no personal fork of username/repoA. I think in this case we should offer to fork the repo and then push to user/repoA even though the user could push to organisation/repoA.

Note: I frequently re-fork repos as a way to navigate to my fork and it doesn't do any harm. So at least it is "safe" to re-fork many times, but still silly.


localhost:6000

Request type

[x] Please close this issue, I accidentally submitted it without adding any details
[x] New documentation
[x] Correction or update

Details

localhost:6000

next


Fix pre-commit regex

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 99.84%. Comparing base (0122962) to head (c5dbb16).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop      #65   +/-   ##
========================================
  Coverage    99.84%   99.84%
========================================
  Files           23       23
  Lines         1301     1301
========================================
  Hits          1299     1299
  Misses           2        2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Misleading or not super useful example

[Enter feedback here]
This example
this is misleading or maybe just a bad example because CountTime isn't used at all in the query.  Yeah, it calculates CountTime so people can see how to do it, but, I would think the query writer could be a little more, creative, I guess, by maybe adding a WHERE clause and say WHERE CountTime > 2 or something like that.
With Temp AS (
SELECT
COUNT(DISTINCT Time) AS CountTime,
Value,
DeviceId
FROM
Input TIMESTAMP BY Time
GROUP BY
Value,
DeviceId,
SYSTEM.TIMESTAMP()
)
SELECT
AVG(Value) AS AverageValue, DeviceId
INTO Output
FROM Temp
GROUP BY DeviceId,TumblingWindow(minute, 5)

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: f0ebe9e5-9a83-b5e6-5dd6-34c841cae021
Version Independent ID: 8c19161b-2eef-ecc9-37fe-5e665da9b2cb
Content: Common query patterns in Azure Stream Analytics - Azure Stream Analytics
Content Source: articles/stream-analytics/stream-analytics-stream-analytics-query-patterns.md
Service: stream-analytics
GitHub Login: @AliciaLiMicrosoft
Microsoft Alias: ali

@markjoh
Thanks for your feedback! We will investigate and update as appropriate.

@markjoh
Thanks for bringing this to our attention.
I'm going to assign this to the document author so they can take a look at it accordingly.

The article has been updated. It will go live around 3 PM PST today.
#please-close


upcoming: [M3-7986] - Fix & Improve Placement Groups feature restriction

Description 📝
This PR improves and simplifies the logic for allowing users to access the Placement Group feature in Cloud Manager.
It originated from the omission to include the "Placement Group" capability in the restriction logic (only the feature flag was used).
I intentionally avoided using isFeatureEnabled, both because it is currently a candidate for refactor, and the fact the hook is clean, concise and isolated to the feature (clear, dedicated naming convention).
Changes  🔄

Create a new hook to check that both the flag is on and the account capability is returned for the current user
Implement hook in

Primary Routes
Placement Group feature
Linode Create flows (v1 & v2)
Primary Nav
Create Menu
GoTo menu

Remove unnecessary checks since the feature is gated properly
Complement coverage

Preview 📷
No visual change as a result of this PR.
How to test 🧪
Verification steps

Using alpha and admin.dev, add/remove the placement-group customer tag to check the feature availability
Toggle the placement-group customer tag to check the feature availability (should be no regression)

As an Author I have considered 🤔
Check all that apply

[ ] 👀 Doing a self review
[ ] ❔ Our contribution guidelines
[x] 🤏 Splitting feature into small PRs
[x] ➕ Adding a changeset
[x] 🧪 Providing/Improving test coverage
[ ] 🔐 Removing all sensitive information from the code and PR description
[x] 🚩 Using a feature flag to protect the release
[x] 👣 Providing comprehensive reproduction steps
[ ] 📑 Providing or updating our documentation
[ ] 🕛 Scheduling a pair reviewing session
[ ] 📱 Providing mobile support
[ ] ♿  Providing accessibility support

Changes look good. e2e failure might be real

@bnussman-akamai yeah add to mock the account - fixed now


dash.docset. settings do not work

I added this to my settings.js file:
  "dash.docset.typescript": [
    "typescript",
    "javascript"
  ],

Assuming that it will only search these two docsets when you run the Dash hotkey in .ts files but that did nothing. It still searches through all these docsets:

Is this a bug? Or I did something wrong?

Hi @nikitavoloboev
I also can reproduce your issue. I threw a question to the Dash developer regarding this. Will keep you posted.
Thank you

@nikitavoloboev
Got this answer from author of Dash

if you send "typescript" (e.g. for a "ts" file) anything TypeScript-related gets activated.

He also mentioned that there is a workaround for this issue by adding keyword exact. Will investigate this.
I also agree that it is not supposed to open other docset outside the configuration.

This still fails for me @deerawan
My settings:
  "dash.languageIdToDocsetMap": {
    "typescript": ["typescript", "javascript"]
  },

I go to .ts file and activate extension.

Result:

@deerawan Any way to know why this happens? Did I do something wrong?


iOS markers api (rebased)

Rebase of https://github.com/tangrams/tangram-es/pull/1044

@blair1618 , @msmollin , comments have been addressed, please give this a last eye and merge when ready.

LGTM - waiting on @blair1618 before merging.


This way all code will stay looking the same and developers won't have to manually run the command before committing.

I haven't done it, but it should be fairly easy. In the CI pipeline we run eslint or prettier. I believe both return an exit code of 0 if nothing needs to change or 1 if they do. Based on that we post to github to add a status to the ref on repo (https://developer.github.com/v3/repos/statuses/#create-a-status).


Renames many things

These are meant for the 3.0.0 development branch, since they break things.
Basically this renames a tonne of things that either no longer make sense, or never did :sweat_smile: Updated the docs and tried to add comments where possible.

These are some pretty big changes to the way we name things – probably best to get input from @Moya/contributors

I definitely prefer the wording here.

Cooool, thanks! I'll give some time for others to review, too.

Looks good to me. :+1:
I particularly like these changes -
 `MoyaProvider.MoyaEndpointResolution` to `MoyaProvider.RequestClosure`
 `MoyaProvider.endpointResolver` to `MoyaProvider.requestClosure`

So much clearer!

will need a rebase, but good to merge from me

:+1: Good for me :)

OK, will merge when I get a chance.

OK, so this has had master and 3.0.0-develop merged in. We should be good to go to merge this – once we do, I don't see too much stopping us from releasing 3.0.0 :tada: If anyone wants a final review, now's the time :smile:

:metal: Nice work. The API is much clearer with these changes.


FragmentManager is already executing transactions

this happens when asking permission in fragments onViewCreated method .
adding some delay fixes the issue , its because , we trying add fragment when fragmentManager is already in execution .

The same issue.

faced this issue too

https://github.com/florent37/RuntimePermission/pull/30
fixed this issue , please check the pr

RxPermissions already got over this issue here https://github.com/tbruyelle/RxPermissions/issues/214

As far as I understand, the main idea is to use getSupportFragmentManager if the user called permissions from an activity and to use getChildFragmentManager if it was called from a fragment. Removing commitNowAllowingStateLoss is just a workaround.

I face this similar problem when I was requesting the system permission e.g WRITE_SETTINGS and was having a call to code which wrapped inside ask permission on the onResume of a fragment that causes this exception to occur. Just make sure about the call you are making inside onResume in your code.


Missing method implementation on X.F.M. v 71

Version Number of Plugin:2.3.6
Device Tested On: Android 7.1, 8, 9.
Simulator Tested On: only physical devices
Version of VS: 16.3.7
Version of Xamarin:<IP_ADDRESS>7
Versions of other things you are using:
Xamarin.Firebase.Messaging v71.1740.0
Xamarin.Forms v<IP_ADDRESS>
Android platform packages v28.0.0
CrossFirebasePushNotification.Current.Subscribe throws missing method exception.

Duplicate of #251

Please test this beta and let me know if solves your issues: https://www.nuget.org/packages/Plugin.FirebasePushNotification/3.0.1-beta


[spike] DCOS-46215: Error requesting UI Configuration message email is shown as {0}

Change the error message function in order to show the email.
Closes https://jira.mesosphere.com/browse/DCOS-46215
Testing

In proxy.dev.js use an outdated cluster url and run npm start.
Verify that the email is shown on the error page

Trade-offs
The message appears in English, even when the language is changed to Chinese, I am not sure why.
Dependencies
https://github.com/mesosphere/dcos-ui-plugins-private/pull/926
Screenshots
Before

After

Fixed commit message, please re-review.

Thank you, @pierrebeitz .

filcts, no pressure

No conflicts anymore, let's see.

Rebased.

:tada: This PR is included in version 2.81.5 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

The functionality is broken, I will create a new PR soon.


Fix illuminance for Aqara P1 motion sensor on initial pairing

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

1 of 4 (25.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.03%) to 80.307%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

zhaquirks/xiaomi/aqara/motion_ac02.py
1
4
25.0%

Totals

Change from base Build<PHONE_NUMBER>:
-0.03%

Covered Lines:
5607

Relevant Lines:
6982

💛  - Coveralls

Codecov Report

Merging #1621 (913ee1f) into dev (71c95b7) will decrease coverage by 0.03%.
The diff coverage is 25.00%.

@@            Coverage Diff             @@
##              dev    #1621      +/-   ##
==========================================
- Coverage   80.33%   80.30%   -0.04%
==========================================
  Files         232      232
  Lines        6978     6982       +4
==========================================
+ Hits         5606     5607       +1
- Misses       1372     1375       +3

Impacted Files
Coverage Δ

zhaquirks/xiaomi/aqara/motion_ac02.py
57.14% <25.00%> (-2.48%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 71c95b7...913ee1f. Read the comment docs.


Problems with removed vertices at tile borders

I've been investigating problems with incorrect pathing, e.g.

After some digging, I've found that the root cause is the logic that removes vertices on tile borders while building polymeshes.
There is a parameter that controls the max edge length while tracing contours, it correctly creates 'small' areas:

When later poly mesh is built, it builds a triangulation for each contour. However, there is a piece of code (see RcMeshs.BuildPolyMesh) that marks vertices on the tile borders for removal:
                    if ((cont.verts[v + 3] & RC_BORDER_VERTEX) != 0)
                    {
                        // This vertex should be removed.
                        vflags[indices[j]] = 1;
                    }

As a result, recast creates a huge polygons on tile borders:

Presumably this is done for better tile stitching? However, it creates problems for pathfinding (since the pathfinding considers edge midpoints, it thinks that going 'around' this polygon is 'shorter' than crossing long edge midpoint).
Removing this vertex removal bit fixes the triangulation (and consequently pathfinding), and doesn't seem to break stitching in my limited testing so far:

However, I don't quite understand all implications. I suspect this part is directly ported from recast?

There are parts that I also haven't fully understood, so it's difficult for me to respond.
Firstly, the part that I don't understand from the perspective of porting is the technical porting.


Strange Frame-set Behavior when pulling multiple streams (depth,infrared) simultaneously

| Camera Model                       | D435 |
| Firmware Version                   | <IP_ADDRESS> |
| Operating System & Version |  Raspbian Buster  |
| Kernel Version (Linux Only)    |  (5.4.79-v7l+)                                         |
| Platform                                 |Raspberry Pi|
| SDK Version                            |  { legacy / 2.. }                          |
| Language                            | C++|
Issue Description
The main issue that we have been encountering, is correctly streaming both infrared streams as well as the depth stream. What seems to be happening is that the frames are not packaged into frame sets correctly. When trying to pull a frame set from the pipeline (which as far as I understand synchronizes the infrared and depth frames into frame sets under the hood), the result ends up pulling depth and infrared images with the same timestamp as separate frames. To best describe the problem, I have attached some simple code below, and will demonstrate some outputs showing the symptoms of the issue.
The code below is a simple frame drop tester that is meant to pull 1000 frames, then as a post processing action, print out the frame # and the frame timestamp:
int main(int argc, char * argv[]) try
{
    // Create a Pipeline - this serves as a top-level API for streaming and processing frames
    rs2::pipeline p;

    rs2::config cfg;

    int width = 848;
    int height = 480;
    int fps = 30;

    cfg.enable_stream(RS2_STREAM_DEPTH, width, height, RS2_FORMAT_Z16, fps);
    cfg.enable_stream(RS2_STREAM_INFRARED, 1, width, height, RS2_FORMAT_Y8, fps);
    cfg.enable_stream(RS2_STREAM_INFRARED, 2, width, height, RS2_FORMAT_Y8, fps);

    rs2::pipeline_profile profile = p.start(cfg);

    std::vector<int> frameNums;
    std::vector<float> timeStamps;

    for (int i = 0; i < 1000; i++)
    {
        // Block program until frames arrive
        rs2::frameset frame = p.wait_for_frames();

        frameNums.push_back((int)frame.get_frame_number());

        timeStamps.push_back(frame.get_frame_metadata(RS2_FRAME_METADATA_SENSOR_TIMESTAMP));

    }
    p.stop();

    for (int i = 0; i < (int)frameNums.size() - 1; i++){
        std::cout << frameNums[i] << " ";

        std::cout << timeStamps[i + 1] - timeStamps[i] << std::endl;
    }

    return EXIT_SUCCESS;
}
catch (const rs2::error & e)
{
    std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    " << e.what() << std::endl;
    return EXIT_FAILURE;
}
catch (const std::exception& e)
{
    std::cerr << e.what() << std::endl;
    return EXIT_FAILURE;
}

The expected output (for this FPS) should be numbers 1-998 (as it is taking the difference in time stamps) with a number around 34000 (34000 microseconds) which is the appropriate difference in time stamps for this FPS.
Indeed, when only running one stream (commenting out the infrared stream declarations), this is exactly the output.
When running multiple streams, it seems as though the frames were not correctly grouped into frame sets, and when pulling them from the pipeline, frames with the same timestamp were pulled several times (not necessarily only 3 times). Further investigation on the github support suggested rebuilding with -DFORCE_RUSB=true instead of -DFORCE_LIBUVC=true.
Once this was done, the issue was still not resolved, and trying to pull the timestamp data yielded the result: "Realsense Error calling rs2_get_frame_metadata(frame:0x16b4848, frame_metadata:Sensor Timestamp): metadata not available"
Note that this error did not occur with LIBUVC. Regardless, the program can still function with only the frame counter metadata and not the timestamps, so the lines related to time stamps were commented out. The resulting output is as follows:

While running 848x480 at 30 FPS
and

While running 848x100 at 300 FPS
Note the doubled and occasionally tripled frame number repeats (should be a new frame number each time as the frames are coming in from the hardware)
The goal of the project is to be able to pull 3 synchronized frames from the left and right IR imagers, and the depth streams and analyze the video. Clearly, the pulling of the frames is working incorrectly in the test script supplied above.
Essentially the two issues that I would like solved is the extremely strange and inconsistent behavior of the frame sets shown above, and the inability to pull timestamps after switching to RUSB_BACKEND.
Some important notes are that the script supplied above works perfectly fine on my Mac OS as well as my colleagues Mac OS. This leads me to believe that it is a problem with the Linux architecture of the pi. To be clear how I installed the librealsense on the pi, the installation guide used is linked below, created by ACROBATIC:
https://github.com/acrobotic/Ai_Demos_RPi/wiki/Raspberry-Pi-4-and-Intel-RealSense-D435
(The python stuff is unimportant since I am writing in C++)
(Upon rebuilding, instead of ACROBATIC's use of LIBUVC, the RUSB_BACKEND flag was used instead)
Any support on this issue would be greatly appreciated

Hi @Decerguyer
The depth is perfectly aligned to the left infrared sensor by default.  The depth origin point of the camera coordinates is the center of the left IR sensor.
https://dev.intelrealsense.com/docs/tuning-depth-cameras-for-best-performance#section-use-the-left-color-camera
The same frame number may occur more than once if a frame is dropped, causing the SDK to return to the last good frame from that stream.
https://github.com/IntelRealSense/librealsense/issues/1686#issuecomment-388606174
If you are generating depth and IR for the purposes of side by side analysis on a video: since left IR is perfectly snchronized with depth by default, it may be worth dropping right IR (index '2') from the script and just having depth and left IR side by side with the assumption that they are synced (as they are generated by the same sensor component in the camera).  Having one less active stream would also reduce the processing load on the Pi.
My understanding is that building with -DFORCE_LIBUVC=true is not wrong, just deprecated (a feature that may be removed at an unknown future date, but is not always removed).  So if it works for you (and the Acrobotic guide recommends it too) then by all means use it instead of RSUSB.

MartyG,
Thank you for the quick response.
Testing with only the left imager ('imager 1') produces the following result:

This time we got up to frame number 503 instead of 998.
Now it may be worth noting that manually timing the script called for a runtime of about ~17.6 seconds which is around half of the necessary time to record 1000 frames at 30 FPS (33.3 seconds). Perhaps every other frame is an infrared and the remainder depth and it is recording half the number of images? But then it wouldn't make sense why on MacOS this problem didn't occur as it ran with no doubled frame numbers (up to 998) and took the full ~33.3 seconds to run.
That being said there are still some frame numbers with a missing pair (assuming the above assumption is correct), which I suppose could be the result of frame drops. Even if The above assumption is incorrect, that would mean even worse performance regarding frame drops. Is there any way to prevent these frame drops? It is unlikely that it is due to any post processing as the frame metadata is immediately being saved to a vector on the stack and discarded.
Frame drops were a serious problem on our Mac OS systems, especially at higher resolutions and FPS, and a huge reason why coding has been done directly on the Pi (which is the end goal anyhow). For one stream only, the Pi was able to outperform our Macs, running any FPS with the allowed resolutions, which we thought was almost certainly an issue of bandwidth. That is partially why it is so surprising to see it happening on the Pi on not the Macs for the specified setting.
As a side note: Is there any reason why timestamps are inaccessible using RUSB instead of LIBUVC?
Thanks

It sounds as though you are using a Raspberry Pi 4 if you are able to set the stream to 848x480 and 30 FPS (as Pi models before 4 only have USB 2.0 instead of USB 3).  Can you confirm if this is correct, please?
I have not seen frame-number checking performed using your method before, so it is difficult for me to debug as I do not have a reference of a similar project to compare it with for best-practices.
A typical way to check frame numbers would be to record a bag file in the RealSense Viewer program using the Record button at the top of the options side-panel and then check the frames after the bag file has been recorded.
You could also open the debug console of the RealSense Viewer by left-clicking on the ^ arrow in the bottom corner of the Viewer window and then start the depth and IR stream and observe the 'Frame Drops Per Second' panel in real-time.

If you would prefer to program your own solution, it may be worth accessing and printing the frame counts of the multiple streams using callbacks, as the C++ SDK example rs-callback does.
https://github.com/IntelRealSense/librealsense/tree/master/examples/callback
You should be able to access hardware metadata with the RSUSB installation method,.  I researched the subject and was reminded of a case from August 2020 with similar circumstances to yours (a Pi, Raspbian Buster and RSUSB = On.  They were not able to receive hardware metadata either and switched to a laptop in the end.
https://github.com/IntelRealSense/realsense-ros/issues/1324

Thank you for the response @MartyG.
First off I would like to confirm that yes, I am using a raspberry pi 4, with USB 3.2.
Secondly, I took your advice of recording and inspecting on the viewer and it showed some interesting results. Perhaps these results will better illuminate the problem we are having as it is quite specific.
Below are 2 bag inspections, 2 images recorded using the MacOS, and 2 recorded using the Raspberry Pi 4. Notice how in the first two images, the MacOS Infrared and Depth inspections, the frames are aligned. The timestamp of the bag file are aligned for both the infrared and depth streams, and the timestamps of the specific frames are the same. In the two images showing the results from the Raspberry Pi, the streams seem to be asynchronous, with the depth stream lagging behind in both the bag file timestamps, and the actual frame timestamps.

Infrared Stream recorded on MacOS

Depth Stream recorded on MacOS
Again, the timestamps are aligned as the frames are arriving at the same time. Keep in mind everything was recorded using the realsense viewer.
Below appears the data from the Raspberry Pi streams:

Raspberry Pi Infrared Stream

Raspberry Pi Depth Stream
It is clear from the last two images that there is some sort of problem with the synchronization when using the pi, as the time stamps do not line up as opposed to when using MacOS.
Both recordings were done using the viewer, and thus I expected similar results. But based on the data it is clear why my script posted earlier was behaving in the way that it was. That being said, I would like to find a reason, or a fix, for this problem so that I can get on with the end goal.
I want to be able to pull the infrared(1) and matching depth stream and be able to pull data simultaneously in the form of frame sets for image analysis. Given the unpredictable nature of the streams on the Raspberry Pi as demonstrated above, this is not possible at the moment. How can I fix this?
Thank you for the responses

I do not have knowledge of metadata operation on Raspberry Pi and Raspian to be able to suggest a corrective course of action.  As Linux kernel and metadata handling improvements were included in the new 2.41.0 librealsense released today, if you are not dependent on Raspian then perhaps you could install Ubuntu as the OS on the Pi and build librealsense from source code with kernel patching (i.e libuvc backend /RSUSB is false).
https://github.com/IntelRealSense/librealsense/wiki/Release-Notes#bug-fixes-and-enhancements

Thank you MartyG.
Since I am not dependent on Raspian, I will try to switch to Ubuntu and try your suggestion.
I will update this thread once I have some results with the new OS.
Thanks again.

I have since tried two versions of Ubuntu, and it had some mixed results.
First, I installed the most recent 20.10 with kernel 5.8. This seemed to fix the synchronization problem between the depth and infrared stream. It did not, however, allow timestamps to be read back from frames.
I then switched to Ubuntu 20.04 (since it is an LTS version) with kernel 5.4, which is supposed to be supported by the most recent Realsense SDK. I used the LTS kernel patch before building. The synchronization problem was again resolved, but frame metadata could still not be accessed. I did not build using any special flags such as LIBUVC or RSUSB since I was under the impression that there were kernel patches that fixed the issue.
The main point here being that the depth and infrared streams work as they should, but timestamps are still unavailable (both of which are required for this project).
Thanks

Apologies for the delay in responding further, as I was carefully considering your case.
Would it be possible please to turn off global time in your C++ script by setting RS2_OPTION_GLOBAL_TIME_ENABLED to false and see if that makes a difference?
https://github.com/IntelRealSense/librealsense/issues/7614#issuecomment-713680733

MartyG,
I apologize, I need to make a correction. Upon further review it seems as though the kernel patch never actually executed. I received a fatal error "fatal: You need to specify a tag name". Looking through issues on the GitHub, it seems as though the only fix for this is to bypass the kernel completely with RSUSB_BACKEND, which would defeat the purpose of changing to ubuntu LTS.
It should be noted, however, that RSUSB_BACKEND has yielded some success using Ubuntu 20.04. 90 FPS with 848x480 seemed to run smoothly and synchronously. 300 fps with 848x100 (which is fewer pixels per second than 90 FPS with 848x480) does not, and the depth/infrared streams are seemingly independent of each other.
Ideally, I would like to be able to patch the kernel such that RSUSB_BACKEND does not need to be utilized, especially since soon a second camera will be implemented.
Is there any way to fix the patch error I received?
Thanks

I extensively researched your problem with needing patching for hardware metadata but could not find many options.   Here are a couple of suggestions:

Since changing the operating system is a possibility for you: metadata support can be added on Windows 10 without patching, and Windows 10 now has support for installation on Arm devices as of mid 2020.  Given that each installation of Windows 10 needs a licence key though, this may not be a practical solution if you are aiming to develop a commercial product rather than a system you are using internally on a small number of Pi's.

https://docs.microsoft.com/en-us/windows/uwp/porting/apps-on-arm

Switch from Pi 4 to an Nvidia Jetson board (which also uses an Arm processor) and use Intel's L4T script to patch a native (non-RSUSB) build.  A Jetson would also make it easier to use multiple cameras.

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md#building-from-source-using-native-backend

Hi @Decerguyer  Do you require further assistance with this case, please?   Thanks!

Hi @Decerguyer  Do you require further assistance with this case, please?   Thanks!

MartyG,
It is unfortunate that the solutions you have offered do not suit our development goals. That being said, we appreciate your assistance with this matter.

MartyG,
It is unfortunate that the solutions you have offered do not suit our development goals. That being said, we appreciate your assistance with this matter.

You are very welcome @Decerguyer - I am sorry that I could not be of more help on this particular occasion.

You are very welcome @Decerguyer - I am sorry that I could not be of more help on this particular occasion.

Case closed due to no further comment received.

Case closed due to no further comment received.


Introduced new section 'DATA_INTEGRITY' with two initial simple tests

(to be continued)
Adapted to current state of implementation of testsuites management and json preprocessor.

Hi Thomas,
both tests successfully executed with latest sources from develop.
But now TSM_0800-(JSON_DOTDICT)-[GOODCASE] fails.
Bus is healed with https://github.com/test-fullautomation/python-jsonpreprocessor/pull/72
All tests passed now.


Function generator: error in yield statement

Hi,
I am getting an error in yield statement 'Error: expected an assignment or function call' message in following code:
function *g (): Iterable<number> {
	for (let i: number = 0; i < 100; i++) {
		yield i;
	}
}

Is there really any error in my code?
Thanks for your reply,
O.

This looks like an error reported by the compiler rather than TSLint... can you post your full stack trace or console output?

A problem on the TSLint side might be that we target ES5 instead of ES6. I think generators only work in ES6.

When I run tslint main.ts I get following main.ts[3, 3]: expected an assignment or function call, so I think this issue has something to do with TSLint.
I am targetting ES6.
Here is my tsconfig.json:
{
  "compilerOptions": {
    "module": "commonjs",
    "noImplicitAny": true,
    "target": "es6",
    "experimentalDecorators": true
  }
}

Hope this information helps somehow.
Thanks,
O.

@zdychacek Sorry, I didn't mean that you need to be targeting ES6; I meant that TSLint ought to have an option that makes it target ES6 when analyzing your code. I don't know if that's the issue for sure though. Definitely a bug either way, thanks for reporting it!

Seems like we can handle this better even when internally targeting ES5, see the above PR :)

Great, thank you!
O.

the fix will be released in v2.5.2. until then, I think you can avoid the issue with /* tslint:disable */ flags?


On Windows, Yeoman generator should stop with error if your NPM version is too old

NPM < v3 on Windows can't correctly restore the full dependency tree for packages like @angular/*.
If you use these Yeoman generators, you'll eventually end up with an error like:
ERROR in ./ClientApp/boot-client.ts
Module not found: Error: Cannot resolve module 'reflect-metadata' in C:\Users\stevesa\Desktop\spapacktest2\ClientApp
 @ ./ClientApp/boot-client.ts 4:0-27

The underlying cause is NPM not restoring the packages properly, and the solution is to update to NPM 3+. We should prevent things from getting this far by stopping the generator up front if you're on NPM < 3 on Windows.

Note that this will only help partially. Even if your command-line-invoked copy of NPM is 3.x, your VS installation might still invoke NPM 2.x, so you'll still have trouble.
When some version of VS ships that includes NPM 3.x, we might want to check that, if you're on Windows, you have that version of VS (or no version of VS).