sigma vertical coords, and 3D advection

Hello everyone,
I have some main questions about OceanParcels that were not clear for me in the tutorials.

Is the 3D advection already working? In the tutorials I haven't seen any example reading w, only uv.
How does parcels handle the vertical coordinate of the models? Is it possible to use outputs from a terrain-following vertical coordinate model (eg. ROMS)? If not yet, is it going to be implemented soon?
Hope anyone can clarify me :)
Best wishes,
Ana

Hi Ana,
Yes we can do 3D advection using 3D fields, and yes we can read s-levels grids!
For the first point, 3D advection is not super visible within all the 2D examples, but it is fully validated and works well.
For the second point, it does also work and we've validated a few examples, but we don't use it so much here in the lab, so I haven't fully tested for example how it reacts to Fieldset.from_netcdf() function.
So please use your ROMS data and contact us in case of any problem (providing a small dataset! )
Cheers,
Philippe

PS: Are your vertical coordinate moving in time? We can read such coordinates, but Fieldset.from_netcdf does not work in such case for sure. If you need this,  you can create the grid yourself following this example (line 157)
https://github.com/OceanParcels/parcels/blob/master/tests/test_grids.py

But the sea surface level is moving, right?

Of course, yes, you are right. Sigma levels move in time with zeta. I think I am not getting what you mean with this function, line 157. Is this to read the data from roms output file?
And how do I define the depth levels of the sigmas?

Since the depth levels are moving in time, we don't have yet a proper function to read it immediately from the files, but still, moving depths can be used with current version of Parcels.
In function starting from line 157, I show an example with sigma levels. The data are not read from the files, I do define them myself.
About how to define the depth levels of the sigmas, what do yo mean? First is to understand what is your mesh. If you have real sigma levels, it goes like this:
With n levels, the depths are
z = -h + (eta + h) * i / (n-1)
for level i (i ranging from 0 to n-1), with h the bathymetry and eta the sea surface elevation.
(or the opposite, maybe level 0 is at the surface, maybe at the bottom)
Could you check that in your data?
Then we can think how to put that into Parcels

I have the functions to define the depth levels of the sigmas, it is a bit more complex because it allows to refine the mesh near the surface and near the bottom (by changing 2 parameters, theta_s and theta_b. In the simulation I would like to use now, I have increased resolution near the surface (theta_s=6).
z levels, where the vertical velocity is computed z_w are defined like this (matlab code):
cff1 = 1./sinh(theta_s);
cff2 = 0.5/tanh(0.5*theta_s);
sc = ((0:N) - N) / N;
N = N + 1;
Cs = (1.-theta_b) * cff1 * sinh(theta_s * sc)...
+ theta_b * (cff2 * tanh(theta_s * (sc + 0.5)) - 0.5);
hinv=1./(h+hc);
cff=hcsc;
cff1=Cs;
for k=1:N
z(k)=zeta+(zeta+h).(cff(k)+cff1(k)*h).*hinv;
end
(https://www.myroms.org/wiki/Vertical_S-coordinate)
u and v are defined at different levels (z_r) as the tracers.
I will really need your help to be able to do this, because now I wouldn't even know where to start.
Do you think this is going to be easy/possible?

Yes, it is possible.
What you have to do is to create a netcdf file in which you define a depth field of dimensions
(tdim, zdim, ydim, xdim), and then you apply your formula
z[ti, zi, yi, xi]=zeta[ti, yi, xi]+(zeta[ti, yi, xi] + h[yi, xi] ).(cff[zi]+cff1[zi]*h[yi, xi]).*hinv[yi, xi];
(zi, yi, ti, xi) are the indices of the data in your arrays.
What is the dimension tdim?
If it is big, you should maybe put one time step (ti) per netcdf file
When this is done, the Parcels part comes. You can use Fieldset.from_netcdf(...., full_load=True)
This last flag is important. Usually, we don't fully load the data since the beginning, but only the time steps which are currently necessary. All is done under the hood
But this won't work with your z moving in time.

Thank you very much for your reply. My tdim depends on the purpose, but for what I want to do first it would be around 80 days. I have outputs at each 5 days, do you think it is enough to have a credible results?
I will try to do the first part, and I will get back to you after that.

With tdim around 20, you could potentially use this full_load=True. (it depends also on the other dimensions, and the RAM of your computer, but you can definitely try).
This means that it will be easier to work once we have the zdepth in a netcdf file, good news!

Sorry, I meant, 80 time steps, not 80 days (one year). So probably it is too much, then. I should make a netcdf file for each step, in this case? And shall I keep the "time" dimension, anyway?

80 time steps will depend on the xdim, ydim and zdim. We'll try and see.
It doesn't really matter if you put it in one single file or in multiple files.
It is good to have a time dimension and time value attached to your different depths (both if you use a single file or multiple files)

ok. thank you :) I will try and write you back when I have made some progress.

Hi Philippe,
Not sure you're already back, if you can help me with this it would be great! Right now I have no idea what to do next.
Cheers,
Ana

Hi Ana,
I guess the codes thinks you want to give him a vertically adaptive mesh, since you give him multiple files for the depth value. Since the depth is constant in time, you should give only one single file.
Does that fix your problem?
Cheers,
Philippe

Hi Philippe,
Thank you for your reply. I am not sure I am doing your suggestion, I changed for this:
filenames = {'U': {'depth': file1, 'lon': grdfile, 'lat': grdfile, 'data': files},
'V': {'depth': file1, 'lon': grdfile, 'lat': grdfile, 'data': files}}
variables = {'U': 'u',
             'V': 'v'}
dimensions = {'U': {'time': 'scrum_time', 'depth': 'SigmaDepths', 'lat': 'lat_u', 'lon': 'lon_u'},
              'V': {'time': 'scrum_time', 'depth': 'SigmaDepths', 'lat': 'lat_u', 'lon': 'lon_u'}}

filenames output is:
{'U': {'data': ['./ncfiles/roms_avg_Y2000_interm.nc.1'],
'depth': './ncfiles/roms_avg_Y2000_interm.nc.1',
'lat': './ncfiles/roms_grd.nc.1',
'lon': './ncfiles/roms_grd.nc.1'},
'V': {'data': ['./ncfiles/roms_avg_Y2000_interm.nc.1'],
'depth': './ncfiles/roms_avg_Y2000_interm.nc.1',
'lat': './ncfiles/roms_grd.nc.1',
'lon': './ncfiles/roms_grd.nc.1'}}
it looks ok, no?
but I'm still getting the error:
In [16]:     fieldset = FieldSet.from_nemo(filenames, variables, dimensions)
...:
NotImplementedError                       Traceback (most recent call last)
 in ()
----> 1 fieldset = FieldSet.from_nemo(filenames, variables, dimensions)
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/parcels/fieldset.pyc in from_nemo(cls, filenames, variables, dimensions, indices, mesh, allow_time_extrapolation, time_periodic, tracer_interp_method, **kwargs)
292
293         return cls.from_netcdf(filenames, variables, dimensions, mesh=mesh, indices=indices, time_periodic=time_periodic,
--> 294                                allow_time_extrapolation=allow_time_extrapolation, interp_method=interp_method, **kwargs)
295
296     @classmethod
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/parcels/fieldset.pyc in from_netcdf(cls, filenames, variables, dimensions, indices, mesh, allow_time_extrapolation, time_periodic, full_load, **kwargs)
229             fields[var] = Field.from_netcdf(paths, var, dims, inds, grid=grid, mesh=mesh,
230                                             allow_time_extrapolation=allow_time_extrapolation,
--> 231                                             time_periodic=time_periodic, full_load=full_load, **kwargs)
232         u = fields.pop('U', None)
233         v = fields.pop('V', None)
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/parcels/field.pyc in from_netcdf(cls, filenames, variable, dimensions, indices, grid, mesh, allow_time_extrapolation, time_periodic, full_load, **kwargs)
172                 depth_filename = filenames['depth'][0]
173                 if len(depth_filename) != 1:
--> 174                     raise NotImplementedError('Vertically adaptive meshes not implemented for from_netcdf()')
175         else:
176             lonlat_filename = filenames[0]
NotImplementedError: Vertically adaptive meshes not implemented for from_netcdf()
In [17]:

I am also getting an error message when I try to import NestedField:
In [17]: from parcels import  NestedField
ImportError                               Traceback (most recent call last)
 in ()
----> 1 from parcels import  NestedField
ImportError: cannot import name NestedField

could you send me a few (small) datafiles such that I can give it a try?
The second problem (with the NestedField import) must be something with your Parcels version. How did you update Parcels? Maybe part of the code already uses NestedField, but not another?

Regarding the second question, I didn't update. I am using the version I downloaded and this function you sent me:
https://github.com/OceanParcels/Parcelsv2.0PaperNorthSeaScripts
the code was probably updated in between. Is there any way to update, or shall I download the new code and follow the installation instructions again?
I uploaded the files I am trying to use to this directory:
https://meocloud.pt/link/eef0a305-0600-474f-8a7c-837017349702/var/
they are not much small, if they are too big for you let me know, and I'll try to create another with fewer time steps
thank you very much for your help
Ana

Hi Philippe,
Did you manage to try anything with the files I sent you?
Best,
Ana

Hi Ana,
The problem with the NestedField is that you are using my code which import NestedField, but your parcels version does not have them. Just remove the import NestedField line
I've tried to download the data now, but the downloading will last 5hours according to my internet browser. Could you create a netcdf file with a few 10Mb (max 500Mb) ? Thanks!

Hi Philippe,
I kept only 4 time steps and the velocity and grid variables, the file is out.nc in the same directory:
https://meocloud.pt/link/eef0a305-0600-474f-8a7c-837017349702/var/
Thank you very much
Ana

Hi Ana,
I wanted to use the nc files to load them into a Parcels fieldset, but variables are missing.
On your post of Jan 15, you were trying to import variable SigmaDepths, but I don't see it in the files.
I can help you to do the first step of loading the nc files into a Parcels fieldset, but I need for that all the information in the ncfiles. I won't have time to create the depth layers as you were explaining on Dec 11th. Could you please provide me a comprehensive nc files containing the field dimensions?
Thank you!

Hi Phillipe,
You are totally right, I am sorry about the mess. In the process of reducing the size, I forgot to add the variables with the depth of the sigmas. I was about to send you the correct files yesterday, but checking if everything was fine this time, I noticed the files were corrupted and had some weird velocity values. I decided to re-do all of the process and its finally done. I just uploaded a new out.nc
file with (I hope) everything necessary (https://meocloud.pt/link/eef0a305-0600-474f-8a7c-837017349702/var/.
eta_tho is y dimension
xi_rho is x dimension
eta_v is y dimension of meridional velocity
xi_u is u dimension of zonal velocity
s_rho is the number of sigma vertical levels
The variables: - SigmaDepths is the average sigma depths (computed with average zeta (free surface))
- SigmaDepthsVar is sigma depths varying in time (computed with zeta(t))
Thank you very much for your availability.
Hope it works! :)
Ana

Dear Ana,
I have finally found the time to download your data and process them. I'm sorry for the delay. As you've seen, we are very busy with the general development of the code and our own ocean applications and can't afford to spend time on all users applications.
As you will see in the example attached, Parcels works smoothly with the vertical coordinates you provided. Also, in your mesh, u, v, w and tracer fields are not provided on the same grid. It is an example of a C-grid.
Such grid is also processed by Parcels, but this requires some assumption in the indexing of the gridded data. Fortunately it is the case in the data you provided, except that your u velocity has one row too much, v velocity has one column too much, and the tracers and w velocity have one row and one column too much. But the unnecessary data are ignored by the code.
To read more about those issues, please read the following submitted paper (https://www.geosci-model-dev-discuss.net/gmd-2018-339/)
You can see in cell number 5 of the attached notebook that your indexing is the same as in the paper (fig. 2)
Also, I haven't checked in your data what w velocity means.
If increasing w means increasing z, everything is ok. Otherwise (like e.g. in nemo, where w is upward, but when going upward, z decreases), you should use fieldset.W.set_scaling_factor(-1).
So basically, you will need to use only cell 13 of the notebook, but please read all the cells, to understand the structure of the data you are handling.
Final comment, you see in that cell this line fieldset.U.grid.time_full = fieldset.U.grid.time_full.astype(np.float64). This is due to a bug that I've just found and will fix asap...
Test_ana_dataset.pdf
Cheers,
Philippe


[PackageGraph] Diagnose duplicate manifests

The Swift 4 package manager does not have the package identity concept
so it is possible for two packages to have the same name. It may or may
not cause issues in practice but we should diagnose and report such
things.

https://bugs.swift.org/browse/SR-5728
rdar://problem/34008219 [SR-5728]: Manifest duplicate command in 'commands' map

@swift-ci please smoke test

I'll change manifest to packages as that makes more sense. Also, /Foo is the package URL and not the path.

@rballard updated

Approved for the Swift-4 branch.

@swift-ci please smoke test


ECS Rate limiting

Hi!
I am very interested in solution, but have a question: did you experience the rate limiting described in https://github.com/weaveworks/scope/issues/2050 ?

We never experienced any throttling from AWS api service while using this.

Sorry for the late response. I will try the system out and we will see whats up.


Form Timeouts

Is your feature request related to a problem? Please describe.
I want a Confirm field automatically go on with the preselection after 5 seconds without user interaction.
Describe the solution you'd like
Add a timeout config to the Run func
Some pseudo code:
form := huh.NewForm(...)
form.Run(cfg{timeout: time.Duration})

huh.NewInput().
...
Run(cfg{timeout: time.Duration})

If a timeout is set, a countdown starts when Run is called. When it finishes, Run does not block anymore and adds preselections to Values.
As soon as a key is hit while countdown has not finished, count down stops and works as w/o timeout
Describe alternatives you've considered
Additional context

Hey! Thanks for creating this issue. We do actually need something like this as we recently removed timeout functionality for gum and would love to add it back due to the huh? transition.

Hi, I am working on this and I would like a complete test case/example of this.

Hi @Delta456,
We would need to implement something like the following on the form, similar to the other options.
func (f Form) WithTimeout(d time.Duration) Form

Then, when we Run() the form, we should pass a context to the tea.Program and cancel that context (use context.WithTimeout)

This needs to be closed as it is implemented now.


There should be means to continue the test when failure implants fail

Today’s test on a Cummins ISX could not be completed when the fault A implant failed.
In particular there is value in the general denominator test in parts 11 and 12 — skipping fault implants to get to parts 11 and 12 is expected user behavior.

<IP_ADDRESS> Actions
a. Global DM6 [(send Request (PGNPG 59904) for PGNPG 65227 (SPNSPs 1213-1215, 3038, 1706)]).
i. Repeat request for DM6 no more frequently than once per second until one or more ECUs reports a pending DTC.
ii. Time-out after every 5 minutes and ask user “‘yes/no”’ to continue if still no pending DTC; and fail if user says “'no'” and no ECU reports a pending DTC.
iii. Permit user to elect to continue test if a DM6 was not observed.

<IP_ADDRESS> Actions
a. Global DM12 [send Request (PG 59904) for PG 65236 (SPs 1213-1215, 1706, and 3038)] to retrieve confirmed and active DTCs.
i. Repeat request no more frequently than once per second until one or more ECUs reports a confirmed and active DTC.
ii. Time-out after every 5 minutes and ask user “‘yes/no”’ to continue if there is still no confirmed and active DTC; fail if user says “'no”' and no ECU reports a confirmed and active DTC.
iii. Permit user to elect to continue test if a DM12 DTC was not observed.

<IP_ADDRESS> Actions
a. Global DM12 [(send Request (PGNPG 59904) for PGNPG 65236 (SPNSPs 1213-1215, 1706, and 3038)]).
b. Repeat request until one or more ECUs reports an active DTC.
i. Time-out after 5 minutes and ask user yes/no to continue if there is still no active DTC.
ii. Fail if user says “no” and no ECU reports an active DTC.
iii. Permit user to elect to continue test if a DM12 DTC was not observed.

Utility of a go on button before 100’s of DM6 or DM12 queries run was confirmed.

Add bypass to example instructions for discussion for task 8

Reconsider best methods with multiple fault implants

Work with #1270 .


add example in sudo command

What type of PR is this? (check all applicable)

[ ] ♻ Refactor
[ ] ✨ New Chapter
[ ] 🐛 Bug Fix/Typo
[ ] 👷 Optimization
[ ] 📝 Documentation Update
[ ] 🚩 Other

Description
Related Tickets & Documents
Added to documentation?

[ ] 📜 readme
[ ] 🙅 no documentation needed

[optional] What gif best describes this PR or how it makes you feel?

@bobbyiliev please review I have added example in sudo command


Attendance title to have the same colour as the other titles

@lcoenen @buckwierd
The Attendance title, please make it just like the Payment, Message and Occasion titles? (the colour of it)

This have been fixed.


Caching

Especially on android, having a builtin caching layer would be v. useful.

I totally agree with this. This is way I have created another library for this. Please checkout Fuse.  https://github.com/kittinunf/Fuse.

Ah, I see. Just when I was looking for one, you have created a solution.

The only caveat for you is Fuse is still considered a beta software. It
works but you might need to do an extra work. But if you are bold enough
for Fuel, Fuse is for you.
On Thu, Apr 28, 2016 at 5:46 PM Ismail<EMAIL_ADDRESS>wrote:

Closed #81 https://github.com/kittinunf/Fuel/issues/81.
—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/kittinunf/Fuel/issues/81#event-644134162

I agree. It isn't quite what I want it to be, but:

I plan to add caching only after I get my app to go live, so atm my research into caching is just research
I'll probs stick issues on the Fuse project, and if I feel up to it, try and contribute to it. Or, if it makes more sense, I'll create a wrapper for it.

Also, I have a question for you. Are you planning on making Fuse work with Fuel in the future? Atm, the networking in Fuse is just using java.net.

Fuse can be implemented to Fuel using interceptors, I think.

Fuse seems to be Android only. With Fuel not having that limitation, wouldn't it make more sense to also offer a generic caching solution?

@kittinunf
I've been lurking here for a few months, and I've evangelizing the use of Fuel at Instructure. I was hoping to add caching to an api client I'm writing and it seems like the ideal cache key -> value pair is Request.url -> Response.body. It seems like this sort of thing can't be done using the request and response interceptors right? Because I'd want a request interceptor to create a response do I need to create a special underlying HttpClient?

Please don't use the url as cache key, as it completely breaks the RFC of HTTP Caching (if that's all you do). There are RFC-compliant cache libraries for android already, and those should be used, apart from the fact that android caches.

(reading)

"The primary cache key consists of the request method and target URI.
However, since HTTP caches in common use today are typically limited
to caching responses to GET, many caches simply decline other methods
and use only the URI as the primary cache key."
Seems like they're suggesting using the URL.

HTTP is not "designed" for browsers. It's for agents, including your fridge, crawlers, and people (apart from browsers). They're also not suggesting to use the URL, they note that many caches are, whilst compliant, hostile to caching. For example, iOS by default at most stores one response per url, which means that if you serve html and json from the same endpoint, it will always throw out one of the two 😢 😡
But to respond to the suggestion:

If a request target is subject to content negotiation, its cache
entry might consist of multiple stored responses, each differentiated
by a secondary key for the values of the original request's selecting
header fields (Section 4.1).

Primary cache key is not the only thing that is being cached on. There are secondary cache keys (read: anything in Vary) that must be used to be compliant. If you do not handle this (if you ignore this), you might return a json response when requesting an html page:
GET example.org/my-page
Accept: text/html

GET example.org/my-page
Accept: application/json

If the server responds the first time with the following response:
...
Content-Type: text/html
Vary: Accept

...you probably do NOT want to serve this file when requesting the JSON representation.
https://tools.ietf.org/html/rfc7234#section-4.1

When a cache receives a request that can be satisfied by a stored
response that has a Vary header field (Section 7.1.4 of [RFC7231]),
it MUST NOT use that response unless all of the selecting header
fields nominated by the Vary header field match in both the original
request (i.e., that associated with the stored response), and the
presented request.

The usage of the keyword MUST NOT means that systems will blow up if you don't satisfy this constraint.

Note:
Both Rails and Chrome had this as a bug whereas the incorrect page representation would be served because of shoddy caching 😞

@dustinevan Thanks so much for liking the library!
In my humble opinion, I really feel that Cache is outside the scope of this library. I rather provide HTTP Client that extensible, useful and solid to be used so it is easy to add/extension with some other needs.
I think Caching itself is a tough problem on its own and the last thing I want is
to provide everything under the sun with just one single library.
You might argue that Fuel already has tons of stuff (especially the Fuel's serializable/extension). While that is true, I do feel regret because I think we have too much ground to cover for a project that runs on a voluntary basis.
Having said that, I do have opensource the caching library of my own named Fuse, where I do have such need for that. I extracted it out from the client project and open source it.
Check it if you like, if not, please find some other libraries that suit your needs.
On the contrary, please report it back if Fuel isn't flexible to integrate with other things, for that, we would be happy to fix/improve that for you.

@kittinunf @SleeplessByte Thanks a ton for responding! Good points. Because I'm writing what amounts to a decorator around fuel for a specific rest api, I think I too am going say that caching is outside the scope of my library.
Many thanks for making fuel. Seriously it's so beautiful!

Also, I'd love to contribute if that's ever needed. Honestly, I've never worked on an open source project embarrassing. Do I just fork and try to solve an issue in the backlog? Or do I need to do something else?

jk, I found it. https://github.com/kittinunf/fuel/blob/master/CONTRIBUTING.md


Overseas passports Ethiopia and Somalia amends

PT story: https://www.pivotaltracker.com/story/show/110399636
Updates the address for collecting passports and supporting documents in Ethiopia and Somalia for overseas passports.
Fact check
https://pt-110399636.herokuapp.com/overseas-passports
Expected changes

URL on gov.uk
Address name changes from British Embassy to UK Visa Application Centre

Before

After

I've just rebased this with master and force pushed back to this PR's branch

LGTM :+1:


[FR] More Icons [AWS IVS player, EXPO]

Checklist / 检查清单

[X] No one has submitted a similar or identical feature request before. / 之前没有人提交过类似或相同的功能请求。
[X] This suggestion does not depart from the original intention of LibChecker. / 这个建议不会背离 LibChecker 的初衷。

Enhancement propose / 改进目的
I want to report some libraries and new icons.
libexpo-modules-core.so, it's the Expo umbrella library for Expo's related React Native libraries.

I forget their trademark icons link, did google and found these icons.

libplayercore.so, it's an AWS IVS player, Idk any icons for this. https://github.com/aws/amazon-ivs-react-native-player
https://github.com/aws-samples/amazon-ivs-player-android-sample/blob/master/README.md

libhermes_executor.so, this one should be marked as Hermes but Idk why it's not.

Solution / 解决方案
No response
Additional info / 额外信息
No response

Submit new rules at here


⚠ ClamAV has degraded performance

In 6139c71, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 74 ms

Resolved: ClamAV performance has improved in b1249b5.


Fix for roughness factor issue #251

Proposed fix for issue #251.
Explanation:
In the base class MetalRough2StandardMap RoughnessFactor is statically set to 0.5 since the standard shader does not support this parameter:
// not supported by the Standard shader
public virtual double RoughnessFactor
{
	get { return 0.5; }
	set { return;
}

The inherited class MetalRoughMap does not override this, so we have the same behavior here, i.e. the original RoughnessFactor that was set in the gltf file gets lost.
However, in class MetalRoughMap the default shader is PbrMetallicRoughness. When looking at line 12 in PbrMetallicRoughness.shader  we can see the following:
_Glossiness("Roughness Factor", Range(0.0, 1.0)) = 1.0
This means that

Glossiness/Roughness Factor is supported by this shader
The default value is 1.0

I guess this is the reason why we always get 1.0 in the end, not 0.5 (see description of issue #251).
In my opinion this issue can be resolved by overriding RoughnessFactor in class MetalRoughMap as proposed in this pull request.
However, I'm not sure if we would have to deal with the case that a shader is used which does not support the property (PbrMetallicRoughness is the default shader, but other shaders can be specified through other constructors).

Looks good to me. @stevenvergenz  or @JohnCopicMS do you have any opinions?

Seems good to me as well. I assume when I override the shaders that I am responsible for implementing any properties like this that I care about.

Looking at old pulls.  It looks like this same change got made by someone else.


[BUG] Categorify problems when using num_buckets

@rnyak @benfred
I am updating the Criteo-HugeCTR Notebook example to use the new API: https://github.com/NVIDIA/NVTabular/blob/764b8960f8bb66ec98d785497f62c50c1457efc8/examples/hugectr/criteo-hugectr.ipynb
Previously, I was proc.add_cat_preprocess([ops.Categorify(out_path=stats_path), ops.LambdaOp(op_name="MOD10M", f=lambda col: col % num_buckets)]) for the categoricals.

If try to use LambdaOp with the new API: cat_features = CATEGORICAL_COLUMNS >> Categorify(out_path=stats_path) >> LambdaOp(f=lambda col: col % num_buckets) I get an empty dict when I try to get the embeddings sizes get_embedding_sizes(workflow).

If I try to use num_buckets within Categorify cat_features = CATEGORICAL_COLUMNS >> Categorify(out_path=stats_path, num_buckets=num_buckets) I get an error due to num_buckets being an int (it seems to be a bug not considering the int case)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-18-671dd4884770> in <module>
----> 1 embeddings = get_embedding_sizes(workflow)
      2 print(embeddings)
/NVTA/nvtabular/ops/categorify.py in get_embedding_sizes(workflow)
    387         current = queue.pop()
    388         if current.op and hasattr(current.op, "get_embedding_sizes"):
--> 389             output.update(current.op.get_embedding_sizes(current.columns))
    390
    391             if hasattr(current.op, "get_multihot_columns"):
/NVTA/nvtabular/ops/categorify.py in get_embedding_sizes(self, columns)
    357
    358     def get_embedding_sizes(self, columns):
--> 359         return _get_embeddings_dask(self.categories, columns, self.num_buckets, self.freq_threshold)
    360
    361     def get_multihot_columns(self):
/NVTA/nvtabular/ops/categorify.py in _get_embeddings_dask(paths, cat_names, buckets, freq_limit)
    414         path = paths.get(col)
    415         num_rows = cudf.io.read_parquet_metadata(path)[0] if path else 0
--> 416         if buckets and col in buckets and freq_limit[col] > 1:
    417             num_rows += buckets.get(col, 0)
    418         if buckets and col in buckets and not freq_limit[col]:
TypeError: argument of type 'int' is not iterable

As a workaround for 2, I can build a dictionary specifying the num_buckets for every column, this works, but the embedding sizes are not what I was expecting.

{'C1': (10000000, 512),
 'C10': (10000000, 512),
 'C11': (10000000, 512),
 'C12': (10000000, 512),
 'C13': (10000000, 512),
 'C14': (10000000, 512),
 'C15': (10000000, 512),
 'C16': (10000000, 512),
 'C17': (10000000, 512),
 'C18': (10000000, 512),
 'C19': (10000000, 512),
 'C2': (10000000, 512),
 'C20': (10000000, 512),
 'C21': (10000000, 512),
 'C22': (10000000, 512),
 'C23': (10000000, 512),
 'C24': (10000000, 512),
 'C25': (10000000, 512),
 'C26': (10000000, 512),
 'C3': (10000000, 512),
 'C4': (10000000, 512),
 'C5': (10000000, 512),
 'C6': (10000000, 512),
 'C7': (10000000, 512),
 'C8': (10000000, 512),
 'C9': (10000000, 512)}

It seems that num_buckets being 10M is replacing the cardinality for every column. However, I was expecting that only the columns its cardinality is greater than num_buckets would be limited, and for the rest it would be the same. This is the behaviors I am getting with the old API and Categorify+LamdaOp.

@benfred @albert17 for viz.
Ben, I explained Alberto that it is expected that he is going to see  num_buckets being 10M is replacing the cardinality for every column. He expects that for the low cardinalities, after hashing, the embedding table shape would be (original n_cats, emb_dim) but I think it will be (num_buckets, emd_dim). so for example, say he is hashing column C6 which has only 6 unique features, so cardinality =6. he is hashing it with hash_bucket = 10000000. He expects, the final embedding table shape for C6 would be (6, emb_dim) but he gets (10000000, embd_dim). But this is expected. I dont think there is something wrong here.
What's your opinion? please correct me if I miss something.

@benfred @albert17 for viz.
Ben, I explained Alberto that it is expected that he is going to see  num_buckets being 10M is replacing the cardinality for every column. He expects that for the low cardinalities, after hashing, the embedding table shape would be (original n_cats, emb_dim) but I think it will be (num_buckets, emd_dim). so for example, say he is hashing column C6 which has only 6 unique features, so cardinality =6. he is hashing it with hash_bucket = 10000000. He expects, the final embedding table shape for C6 would be (6, emb_dim) but he gets (10000000, embd_dim). But this is expected. I dont think there is something wrong here.
What's your opinion? please correct me if I miss something.

@albert17 you can also use HashBucket op if you are only doing simple hashing. you can still return emb_size for embedding table, it s now implemented. see the link.
https://github.com/NVIDIA/NVTabular/blob/main/nvtabular/ops/hash_bucket.py#L96-L102

@albert17 you can also use HashBucket op if you are only doing simple hashing. you can still return emb_size for embedding table, it s now implemented. see the link.
https://github.com/NVIDIA/NVTabular/blob/main/nvtabular/ops/hash_bucket.py#L96-L102

We tried HashBucket in the past, but we came up to the conclusion that it was better to use LambdaOp. See discussion here: https://github.com/NVIDIA/NVTabular/issues/402

We tried HashBucket in the past, but we came up to the conclusion that it was better to use LambdaOp. See discussion here: https://github.com/NVIDIA/NVTabular/issues/402

We tried HashBucket in the past, but we came up to the conclusion that it was better to use LambdaOp. See discussion here: #402

@albert17 I check the issue you linked. Are you using LambdaOp(lambda col, gdf: gdf[col].hash_values() % bucket_size) or LambdaOp(f=lambda col: col % num_buckets). These are two different things. The first one will generate hash_values first, end then do modulo operator. You will end up a value between [0, 10M) for each categorical feature if you do the first one. Like in this example:
gdf = cudf.DataFrame({'platform':[3, 3, 12, 12, 1, 2, 0, 1]})

bucket_size = 10000000
f = lambda gdf: gdf['platform'].hash_values() % bucket_size
gdf['hashed_platform'] = f(gdf)
print(gdf)
   platform  hashed_platform
0         3          8043756
1         3          8043756
2        12          6190381
3        12          6190381
4         1          2460029
5         2          3430573
6         0          9140149
7         1          2460029

so using LambdaOp(lambda col, gdf: gdf[col].hash_values() % bucket_size) and using Hash_Bucket(num_buckets=10M) is same.

We tried HashBucket in the past, but we came up to the conclusion that it was better to use LambdaOp. See discussion here: #402

@albert17 I check the issue you linked. Are you using LambdaOp(lambda col, gdf: gdf[col].hash_values() % bucket_size) or LambdaOp(f=lambda col: col % num_buckets). These are two different things. The first one will generate hash_values first, end then do modulo operator. You will end up a value between [0, 10M) for each categorical feature if you do the first one. Like in this example:
gdf = cudf.DataFrame({'platform':[3, 3, 12, 12, 1, 2, 0, 1]})

bucket_size = 10000000
f = lambda gdf: gdf['platform'].hash_values() % bucket_size
gdf['hashed_platform'] = f(gdf)
print(gdf)
   platform  hashed_platform
0         3          8043756
1         3          8043756
2        12          6190381
3        12          6190381
4         1          2460029
5         2          3430573
6         0          9140149
7         1          2460029

so using LambdaOp(lambda col, gdf: gdf[col].hash_values() % bucket_size) and using Hash_Bucket(num_buckets=10M) is same.

I was doing ops.LambdaOp(op_name="MOD10M", f=lambda col: col % num_buckets)
From the past disscusion I understood we wanted to avoid hash_values(), we just want to apply the modulo 10M

@benfred @rnyak Does it makes sense that when applying cat_features = CATEGORICAL_COLUMNS >> Categorify(out_path=stats_path) >> LambdaOp(f=lambda col: col % num_buckets) when I call get_embedding_sizes I get an empty dict?

Following your example @rnyak. It seems like there is some problem with LambdaOp
>>> import nvtabular as nvt
>>> import cudf
>>> gdf = cudf.DataFrame({'platform':[3, 3, 12, 12, 1, 2, 0, 1]})
>>> gdf
   platform
0         3
1         3
2        12
3        12
4         1
5         2
6         0
7         1
>>> dataset = nvt.Dataset(gdf)

# Categorify + LamdaOp
>>> cat_features = "platform" >> nvt.ops.Categorify() >> nvt.ops.LambdaOp(f=lambda col: col % 10)
>>> workflow = nvt.Workflow(cat_features)
>>> wrokflow.fit(dataset)
>>> workflow.transform(dataset)
<nvtabular.io.dataset.Dataset object at 0x7fe2af5ba9d0>
>>> nvt.ops.get_embedding_sizes(workflow)
{}

# Only Categorify
>>> cat_features = "platform" >> nvt.ops.Categorify()
>>> workflow = nvt.Workflow(cat_features)
>>> workflow.fit(dataset)
>>> workflow.transform(dataset)
<nvtabular.io.dataset.Dataset object at 0x7fe2af5bab50>
>>> nvt.ops.get_embedding_sizes(workflow)
{'platform': (6, 16)}

# Categorify + HashBucket
>>> cat_features = "platform" >> nvt.ops.Categorify() >> nvt.ops.HashBucket(10)
>>> workflow = nvt.Workflow(cat_features)
>>> workflow.fit(dataset)
>>> workflow.transform(dataset)
<nvtabular.io.dataset.Dataset object at 0x7fe2c020a610>
>>> nvt.ops.get_embedding_sizes(workflow)
{'platform': (10, 16)}

Using just Categorify or Categorify + HashBucket I get the embedding table sizes; however, if I use Categorify + LamdaOp I get an empty dict.

I have tried using HashBucket, and I am getting the same result than with Categorify.
{'C1': (10000000, 512),
 'C10': (10000000, 512),
 'C11': (10000000, 512),
 'C12': (10000000, 512),
 'C13': (10000000, 512),
 'C14': (10000000, 512),
 'C15': (10000000, 512),
 'C16': (10000000, 512),
 'C17': (10000000, 512),
 'C18': (10000000, 512),
 'C19': (10000000, 512),
 'C2': (10000000, 512),
 'C20': (10000000, 512),
 'C21': (10000000, 512),
 'C22': (10000000, 512),
 'C23': (10000000, 512),
 'C24': (10000000, 512),
 'C25': (10000000, 512),
 'C26': (10000000, 512),
 'C3': (10000000, 512),
 'C4': (10000000, 512),
 'C5': (10000000, 512),
 'C6': (10000000, 512),
 'C7': (10000000, 512),
 'C8': (10000000, 512),
 'C9': (10000000, 512)}

I have tried using HashBucket, and I am getting the same result than with Categorify.
{'C1': (10000000, 512),
 'C10': (10000000, 512),
 'C11': (10000000, 512),
 'C12': (10000000, 512),
 'C13': (10000000, 512),
 'C14': (10000000, 512),
 'C15': (10000000, 512),
 'C16': (10000000, 512),
 'C17': (10000000, 512),
 'C18': (10000000, 512),
 'C19': (10000000, 512),
 'C2': (10000000, 512),
 'C20': (10000000, 512),
 'C21': (10000000, 512),
 'C22': (10000000, 512),
 'C23': (10000000, 512),
 'C24': (10000000, 512),
 'C25': (10000000, 512),
 'C26': (10000000, 512),
 'C3': (10000000, 512),
 'C4': (10000000, 512),
 'C5': (10000000, 512),
 'C6': (10000000, 512),
 'C7': (10000000, 512),
 'C8': (10000000, 512),
 'C9': (10000000, 512)}

yes, this is expected, and there is no problem here.

Following your example @rnyak. It seems like there is some problem with LambdaOp
>>> import nvtabular as nvt
>>> import cudf
>>> gdf = cudf.DataFrame({'platform':[3, 3, 12, 12, 1, 2, 0, 1]})
>>> gdf
   platform
0         3
1         3
2        12
3        12
4         1
5         2
6         0
7         1
>>> dataset = nvt.Dataset(gdf)

# Categorify + LamdaOp
>>> cat_features = "platform" >> nvt.ops.Categorify() >> nvt.ops.LambdaOp(f=lambda col: col % 10)
>>> workflow = nvt.Workflow(cat_features)
>>> wrokflow.fit(dataset)
>>> workflow.transform(dataset)
<nvtabular.io.dataset.Dataset object at 0x7fe2af5ba9d0>
>>> nvt.ops.get_embedding_sizes(workflow)
{}

# Only Categorify
>>> cat_features = "platform" >> nvt.ops.Categorify()
>>> workflow = nvt.Workflow(cat_features)
>>> workflow.fit(dataset)
>>> workflow.transform(dataset)
<nvtabular.io.dataset.Dataset object at 0x7fe2af5bab50>
>>> nvt.ops.get_embedding_sizes(workflow)
{'platform': (6, 16)}

# Categorify + HashBucket
>>> cat_features = "platform" >> nvt.ops.Categorify() >> nvt.ops.HashBucket(10)
>>> workflow = nvt.Workflow(cat_features)
>>> workflow.fit(dataset)
>>> workflow.transform(dataset)
<nvtabular.io.dataset.Dataset object at 0x7fe2c020a610>
>>> nvt.ops.get_embedding_sizes(workflow)
{'platform': (10, 16)}

Using just Categorify or Categorify + HashBucket I get the embedding table sizes; however, if I use Categorify + LamdaOp I get an empty dict.

My understanding, applying Categorify + Lambda op would not change  the embedding table shape the one that you'd get from only Categorify op. Because, Lambda op does not have a link with get_emb_size function and it does not collect stats to do so. However, you should still get an embedding table from nvt.ops.get_embedding_sizes(workflow), because you use Categorify before Lambda op. It should print out an embedding table.

This is how I was using it before
num_buckets = 10000000
proc.add_cat_preprocess([ops.Categorify(out_path=stats_path), ops.LambdaOp(op_name="MOD10M", f=lambda col: col % num_buckets)])

And I was getting the embedding sizes like this
embeddings = [c[0] for c in ops.get_embedding_sizes(workflow).values()]
embeddings = np.clip(a=embeddings, a_min=None, a_max=num_buckets).tolist()
print(embeddings)

And the result was looking as expected

[10000000, 10000000, 3014529, 400781, 11, 2209, 11869, 148, 4, 977, 15, 38713, 10000000, 10000000, 10000000, 584616, 12883, 109, 37, 17177, 7425, 20266, 4, 7085, 1535, 64]

@albert17 with the new api changes you can do something like:
categorify_op = nvtabular.ops.Categorify()
cat_features = CAT_COLUMNS >> categorify_op >> (lambda col: col % num_buckets)

workflow = nvtabular.Workflow(cat_features + other_features)
workflow.fit(dataset)

embeddings = {col: (min(count, num_buckets), dim) for col, (count, dim) in
                            categorify_op.get_embedding_sizes(CAT_COLUMNS).items()}

to achieve the same result. Note that the global 'get_embedding_sizes' from the workflow isn't working anymore because of the lambda op after the categorify - https://github.com/NVIDIA/NVTabular/blob/4606b0c7d8ce81f79175db432ddc429fa4fa0c87/nvtabular/ops/categorify.py#L395-L397 -

We should also implement fixed sized embedding sizes in #194

fixed by https://github.com/NVIDIA/NVTabular/pull/544


dtMonthNum should have the signature Int -> DateTime

From the source code:
{-| Month of a year.
-}
dtMonth : MonthName -> DateTime
dtMonth =
    DTMonth

{-| Month of a year as a number (1 - 12).
-}
dtMonthNum : MonthName -> DateTime
dtMonthNum mon =
    case mon of
        Jan ->
            DTMonthNum (Num 1)

        Feb ->
            DTMonthNum (Num 2)

Shouldn't dtMonthNum be the following instead?
dtMonthNum : Int -> DateTime
dtMonthNum n =
     DTMonthNum (Num n)

Otherwise I don't see the point of it when we have dtMonth.

Thanks for spotting. Embarrassing that I let this one slip through. I've deprecated dtMonthNum as it is less safe than using enumerated types. Don't want to remove it entirely yet as this would be a breaking change, but will do so for the next major breaking change release that may be forced by the core VegaLite API.


SOLR-16967: ConfigSetService.checkConfigExists

This method should not look for solrconfig.xml specifically; the parent directory's existence is fine.
https://issues.apache.org/jira/browse/SOLR-16967

Proposed CHANGES.txt:

SOLR-16967: ConfigSetService.checkConfigExists should not require solrconfig.xml's existence.

Will merge tonight


Automatic Camera Settings

Hello there, is it possible to add Automatic Camerasettings for iso and Brightness and so on?
Thx for Help

As far as I know there is no specific automatic brightness setting in the MMAL interface which is used to control all the camera settings.
Some software (e.g. motion) do have a auto_brightness setting which is used to digitally adjust the data in the frame buffers to give an average pixel level of say 128. However, it is normally recommended not to use this with cameras like the pi module which can automatically set the exposure in the actual camera capture. This maximises the quality as processing after capture reduces the dynamic range.
Of course, I stand to be corrected and if there are additional MMAL settings then I can look into supporting those.


Release 3.1.0

Added

Dispatch 'populated' event when a Select's options are populated. Useful for knowing when it's safe to set the initial selectedID value.

Fixed

Updated npm dependencies.

:tada: This PR is included in version 3.0.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Switched LOD not working again

Godot version
3.5 b1
System information
All
Issue description
I read that Godot 3.5 will support LOD - I was delighted. But apparently this did not apply to switchable LODs. This feature worked in 2.x and already works in 4.x, i can only speculate, perhaps you could have this feature in version 3.5 as well? I humbly ask the development team for this small courtesy, unless of course it will embarrass you and will not be undue impudence on my part.

Steps to reproduce

Create an object;
Set distances;
Rotate the mouse roller.

Minimal reproduction project
No response

There's no LOD in 3.x currently afaik (despite the settings being there).

@lawnjelly why "feature proposal"?
I am not suggesting adding something new. It's there, but it doesn't work. If this is not a bug, then what is a bug?

I am not suggesting adding something new. It's there, but it doesn't work. If this is not a bug, then what is a bug?

Sorry to be more clear, this may be a language thing:

There is a setting that you can see in the user interface as shown in red on your screenshot. But this setting does nothing, it is probably leftover from Godot 2, or left for future expansion.
Code in the user interface is like the tip of an iceberg, it is not the functionality. The majority of the iceberg is under the water surface, and in this case, there is nothing under the surface, the other 99% of the code that does something has not been written.

This is a duplicate of #40784. Indeed there's no implementation behind these settings, they're leftovers from 2.x which never got reimplemented (and they're not easy to reimplement).

I know it doesn't work (I've been waiting for LOD switches since version 3.0 for a very long time). I also understand that the code is not there.
But I checked, this functionality works in Godot 4. Godot 4 is very far from working. Therefore, now when many new developments began to be transferred from 4.x to 3.x, I hoped that this functionality could also be restored. I admit that I can be mistaken, but in theory, switching visibility based on the distance condition is a very simple technology.

But I checked, this functionality works in Godot 4. Godot 4 is very far from working.

Not true (unless you're someone whose computer does NOT support Vulkan) - it does have its issues and porting a big project isn't easy, but Godot 4 definitely works.

Therefore, now when many new developments began to be transferred from 4.x to 3.x, I hoped that this functionality could also be restored.

If all you wanted to do was request a backport of the LOD from 4.x to 3.x, the issue should have been titled accordingly (the current title suggests a regression, which is not the case as it never worked in the first place)

Not true (unless you're someone whose computer does NOT support Vulkan) - it does have its issues and porting a big project isn't easy, but Godot 4 definitely works.

I did not mean that Godot 4 does not start, I mean that it is too unstable for development using it.

If all you wanted to do was request a backport of the LOD from 4.x to 3.x, the issue should have been titled accordingly (the current title suggests a regression, which is not the case as it never worked in the first place)

Apparently I do not understand some of the subtleties. I see the same settings in Godot 2.x, 3.x and 4.x. But the technology only works in 2.x and 4.x.
So you are saying that I need to recreate the issue with a different title?

Apparently I do not understand some of the subtleties. I see the same settings in Godot 2.x, 3.x and 4.x. But the technology only works in 2.x and 4.x.

A setting is not a feature. Here's the implementation of visibility ranges in 4.0: #48847.
This is not trivial to backport, it depends on the renderer which is the main part which is 100% different in 2.x, 3.x, and 4.x. Rewriting the renderer is the main reason why these versions are not compatible with each other. It also mean that renderer-dependent features need to be reimplemented from scratch, and LOD was never reimplemented in 3.x. At this point, it's safe to assume it will not be.

It's clear. Thanks for the explanation. It's very unfortunate. This functionality is sorely lacking.

Apparently there's a draft PR which implements the basic LOD functionality that could be expected from these properties: #53778.
So if it can be finalized, this would be merged instead of removing the properties.


Feature request: Looping animations

First of all thanks for that nice little library! I'm using this to get some simple animations done, but it would be awesome to also get the opportunity to loop animations (e.g. pulsating animations to draw the users attention).

El proyecto se moverá a https://github.com/FormsCommunityToolkit
More options (looping), more types of animations (color), etc.

FormsCommunityToolkit has moved to https://github.com/xamarin/XamarinCommunityToolkit

@jsuarezruiz , will this project receive additional features on future? It would be nice to see a looping (and maybe a Delay) property.

Yes, will include this two options and others.

Hey @fabcall take a look to: https://github.com/jsuarezruiz/Xamanimation/pull/13
Included Delay and RepeatForever properties.

@jsuarezruiz, that sounds amazing! Can't wait to try :-D


Typescript Plugin throws an error

Describe the bug
The Typescript example given in the docs doesn't work. Instead it throws this error:
Plugin typescript does not export a valid JS object with "plugin" function.
I'm just using the built-in typescript plugin.
To Reproduce
Follow the quickstart, use the config from the Typescript example in the documentation

My GraphQL schema:
Not schema related. Try any schema.

My GraphQL operations:

N/A

My codegen.yml config file:

schema: http://localhost:1337/graphql
documents: ./src/**/*.graphql
generates:
  ./src/types.ts:
    plugins:
      - typescript

Expected behavior
The library should work as advertised, doing what it claims to do in the docs.
Please stop with the radical changes. This library was working wonderfully a few months ago. It's eating valuable dollars and time right now.
When I download a library, I expect the docs on the library's website to be for the same version of the current release. However, the docs are clearly out of touch. Notice the Programmatic usage section.
Environment:
Node v10.10.0 on a 2018 Macbook pro
Lib version: 1.0.7, the current release.
Additional context
It was working before 1.0 broke everything.

I ran through everything again from scratch, this time using the cli init command instead of copying the yml file from the Typescript example in the docs. It worked. Seems the docs are out of date...

For reference, here are a few parts of my old workflow that the 1.0.X updates have broken:

The docs for the programmatic api are out of date, making it impossible to use the programmatic api anymore (I assume it actually does work, I just have no way of knowing how)
The new 'core' module lacks a bunch of basic functionality (such as pulling schema from a remote server) that was standard in the old library
The CLI requires a globally installed graphql package. As a rule, I NEVER rely on globally installed packages, and run everything using the locally installed copy via an NPM script. This makes sure my team members stay in sync.

All in all, I feel that 1.0 has been released prematurely.

Hi @robbyemmert ,
A few notes regarding the issues you had:

It looks like something is wrong with the yml you used, I'm not sure what it is, because I couldn't reproduce it, and you didn't provide a repository than reproduces it.
The docs are updated, if there is a mistake with the docs, then that's something else, but it's updated.
The CLI does not require graphql to be globally installed. If you install the codegen as global dependency, than yes, you need graphql to be global as well. We do not recommend to install it globally.
The programmatic API is updated and working, and the docs are updated. If you think that it doesn't work, please open a new issue with more information.

Regarding the API changes:

The API has changed in 1.0, that's why it called 1.0, because we introduced breaking changed, and we follow semver.
The core library has changed because we wanted to make it agnostic, and remove all the NodeJS dependencies. This way it's now possible to run the codegen in browsers, with a minimal bundle size and tree-shaking. That's why the programmatic API has changed as well. And again - only in 1.0, according to the semver standard.

Regarding the output changes:

There was an issue from 7 months ago, and we started to discuss the changes we did in 1.0, there is nothing new in it. https://github.com/dotansimha/graphql-code-generator/issues/713
The new API changes and the breaking changes was announced, in a pinned issue, for over a month. https://github.com/dotansimha/graphql-code-generator/issues/1350
The work on the refactor took over a month, of multiple people, and we did it gradually with alpha and beta versions before the release. We are also doing projects for clients, and we tested it comprehensively before releasing it to the public.
The output is much more optimized and has less issues, and not it's very easy to find issues before they happen.

Also, a note about open-source:
This project is an open-source, and we are a team of developers, putting our best efforts to make this library free, reliable, customizable and easy-to-use. We are constantly upgrading it and adding more plugins and more features. Sometimes things breaks, but that's the beauty of open-source - you can always participate and contribute.
If you think something is not right or not updated - you can always open a PR that fixes it and not just open an issue and complain.
And in a more personal manner - if you thing that we are wasting your "valuable dollars and time" - please do not use this library at all. There are other solution for generating code from GraphQL.
We will always introduce "radical changes" because GraphQL (and software, in general) evolves all the time, and we must stay on top of it if we want this package to remain relevant.
I know not all your expectations are met with the 1.0 release, but my expectations from the community is to take an active part on developing it, and not just complaining about it.

I face the same issue today, just follow the instruction on Github Readme.
It failed when trying to generate the .ts file.
The error message is Invalid Custom Plugin "typescript" and Plugin typescript does not export a valid JS object with "plugin" function.

@dotansimha It will be great to provide a solution to this issue. I am facing the same error using the library out of the box. My codegen.yml is:
overwrite: true
schema: "./schema.graphql"
documents: null
generates:
  ./types.ts:
    plugins:
      - "typescript"

I get the error Plugin typescript does not export a valid JS object with "plugin" function.


Use HTML5 data-* attributes

Can I propose to use HTML5 data-* attributes for configuration? This will make the HTML used to add shareNice in the page valid!

Hello, you are spot on.
I fixed this today I just hadn't flushed my nginx cache :)
See the git commit :
https://github.com/mischat/shareNice/commit/984c2dc5001903a86fa868e6119ef78959a4e8a8
I have also updated the docs on the webpage.
Do let me know if you end up using shareNice. I personally have no way of knowing that people are using it. Given all logs go to /dev/null !


Type inference not working with extends generic argument and keyof and --strictNullChecks

The title of the bug should probably be improved - I was not able to come up with something that encapsulates the issue.
TypeScript Version:  2.1.5, compiling using --strictNullChecks
This issue seems not to be present in the TypeScript playground (2.1.4-insiders.20161201):
Code
interface Foo {
    foo: number|null;
}

export class Base<T extends Foo> {
    public style: Readonly<T>;

    public styleDirect: Readonly<Foo>;

    public reqNumber(a: number) {
    }

    public reqNumberNull(a: number|null) {
    }

    public method() {
	// works
        this.reqNumberNull(this.style.foo);

	// works
        this.reqNumber(this.styleDirect.foo || 0);

        // does not work
        this.reqNumber(this.style.foo || 0);
    }
}

Expected behavior:
All three method calls should compile correctly.
Actual behavior:
test.ts(26,24): error TS2345: Argument of type 'T["foo"] | 0' is not assignable to parameter of type 'number'.
  Type 'T["foo"]' is not assignable to type 'number'.
    Type 'number | null' is not assignable to type 'number'.
      Type 'null' is not assignable to type 'number'.

This issue is still present in version 2.2.1-insiders.20170217

Running into something similar (i think), on 2.2.2. Below is some sample code that capture the issue I was seeing with generic type inference when dealing with a generic union type.
// some complex generic union type alias
type ArrayOrValue<T> = Array<T> | T;

// sample function to consume the complex type above
function foo1<T>(x: ArrayOrValue<T>) {
  return x;
}

// some nullable input
let bar: string | undefined;

// bar being inferred as string instead of string | undefined
foo1(bar);
// but there is no problem if the generic typing is explicit
foo1<string | undefined>(bar);

// simpler version of foo1
function foo2<T>(x: T) {
  return x;
}

// these calls have no problem with type inference
foo2(bar);
foo2<string | undefined>(bar);

@ahejlsberg looks like this is one case that was not handled by https://github.com/Microsoft/TypeScript/pull/15576. any thoughts on this?

The problem here is Readonly<T> - TS can't perform the higher-order reasoning to know what comes out the other side is a narrowable thing.

Turns out this issue has been fixed since 4.3.


Pouch / Cloud Planning integration (WIP)

switch from protocol override (which led to weird Storage Keys like:
pouchdb://arcs-storage.firebaseio.com/...
stop using private methods in Planificator.  Instantiate a separate
storageprovider base.
revamp storagekey construction.

This is reviewable, but includes other PRs that can be split off.

outdated.


[Bot OAuth] Bot auth in AuthUrl

Added, still needs testing

For the moment, will start work on issue #41


How to authenticate?

I've just updated to the latest version and have absolutely no clue how to authenticate. I was previously using username/password.
The integration loads and "Requires Configuration".
The instructions state:
Create an account on Husqvarna developers portal. Connect The Authentication API and Automower Connect API. As redirect-URI set https://my.home-assistant.io/redirect/oauth. If you haven't set up My HomeAssistant already. Enter this URL for your instance http://192.XX.XX.XXX:8123/
It asks for:
Name:
OAuth Client ID:
OAuth Secret:
For Name I put the name of the application created in Husqvarna.
OAuth Client ID: I tried Application Key
OAuth Secret: I tried Application Secret
It just returns "Cannot authenticate."
A comment 6 days ago states "You have to add client_id and client_secret in your configuration.yaml". Are client_id and client_secret the application key and application secret? What should they be added as and how?

@MRobi1, I got the same issue.
What I just did was:

Delete the existing integration
Delete the existing oauth token (https://github.com/Thomas55555/husqvarna_automower#remove-credentials)
Reboot HA
Installed the integration
Re-entered the following (created at the developers page of husqvarna)
Name
ApplicationKey
ApplicationSecret

And it works again.

Can you enable debugging, like described in the readme and report the logs?


Add MSVC install instructions

I've been using these SDL2 bindings via glium-sdl on Windows. This PR adds instructions to the README.md that work for me. I based the instructions on the minGW instructions. I use rustup so I don't know if the paths are exactly correct for the other installation options but like I said, I based them off the existing windows instructions. I use a LIB env variable so I know that part of the instructions is correct.
For me the location involving AppData was wrong so I updated that too. I don't know if that means rustup changed the path at some point or what.
Furthermore, I'm not seeing the segfault during init that some folks have seen on Windows. Perhaps because it's all built with msvc or maybe one of the dependencies just handles that case.

My PR here: https://github.com/AngryLawyer/rust-sdl2/pull/550 has a similar intent.  I also removed the warning claiming it would crash on MSVC edify was missed here, but I like your changes to the rustup paths as I myself don't use rustup so I didn't know any better.  Recommend merging 550 and 551 to have both the warning removal and updated rustup paths.

Thanks. I made the changes. Let me know if I should change anything else.

Perfect!


Windows local data gets full

Basic information

My operating system:  Windows 10
My MZmine version: 3.0.0

What happened
The main issue is that it appears that the spectral files get uploaded to the temp area in your local app part of windows and then is not cleared when mzmine is shut down. I guess this is to help with memory management but is there any way to clear the cache once mzmine is closed?

Ah is it so that if a different user restarts mzmine the data will not be cleared from other users cache?

This is the case. We will add a notification at startup so that people change the temp folder to a public directory with enough space.
See #759

Thank you!
Sent from my Galaxy
-------- Original message --------
From: Robin Schmid @.>
Date: 29/06/2022 18:41 (GMT+02:00)
To: mzmine/mzmine3 @.>
Cc: Alex Dickens @.>, Author @.>
Subject: Re: [mzmine/mzmine3] Windows local data gets full (Issue #758)
This is the case. We will add a notification at startup so that people change the temp folder to a public directory with enough space.
See #759https://github.com/mzmine/mzmine3/pull/759
—
Reply to this email directly, view it on GitHubhttps://github.com/mzmine/mzmine3/issues/758#issuecomment-1170141971, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALLFFCESR2LJISIZWNLCYEDVRRVEDANCNFSM52EPPWXA.
You are receiving this because you authored the thread.Message ID: @.***>


How to obtain the plugin address

Greetings, esteemed developer. I have thoroughly utilized your creation with utmost proficiency. Regrettably, due to my inability to acquire access to an account for gpt4.0, I find myself uninformed regarding the whereabouts of the plugin. May I inquire whether you possess a viable resolution or perhaps the means to effortlessly synchronize the address of the OpenAI plugin? I express my gratitude in advance.

here you can find plugins json urls  some will work most will not work
https://chat-gpt-plugins.getit.ai/gpt-plugins


RB-19  Lombok works incorrect

Doen't work @RequeredArgsConstructor and @Data in some places

Fixed


Added MySQL support back to ratings microservice

Release note:

NONE

Hi @ErikWittern. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

don't merge yet

/ok-to-test


Provide feedback if "place order" is not available yet

e.g. "shipping address must be filled" or "email address must be filled"

Highlight inputs with invalid/missing data and focus on first field with error

Included in the pull request: #70

Test result:

The behavior is a bit irritating: the forms are considered valid when everything is entered, regardless if "update" has been clicked. so when I enter everything but for the email address forget to click "update", I get no validation message. I think the best would be to trigger the "update" actions of all changed forms when clicking "place order". maybe in that order:

validate
if valid and unsaved, trigger "update"
actual place order

Another pull request which fixes above reported issue: https://github.com/hyva-themes/magento2-hyva-checkout/pull/81


sysctl: unknown old 'hw.optional.arm64' when building for iOS

Steps to reproduce
The error occurs when executing
flutter build iOS
This is very specific to my project, but I have a complete log from flutter build ios -v, and I believe I know the exact error message:
sysctl: unknown oid 'hw.optional.arm64'
The command: flutter build ios
I have tried
flutter clean
flutter pub get
pod repo update
pod deintegrate
pod install
Removing ios/Pods, ios/Podfile.lock, DerivedData directory
Machine model:
MacBook Pro 2020 Sonoma 14.6.1
Actual results
flutter_build.log
The generic error message is:
Error (Xcode): Error parsing assemble command: your generated configuration may be out of date. Try re-running
'flutter build ios' or the appropriate build command.

The relevant (I think) snippet from the log is below but I believe the actual error is this:
 sysctl: unknown oid 'hw.optional.arm64'
Would appreciate any suggestions as to what the cause of this error might be.
Logs surrounding error message (full log file uploaded here as attachment):
`/bin/sh -c /Users/gudmhb/Library/Developer/Xcode/DerivedData/Runner-abizgewvidhcnpalwierdzgwcuyl/Build/Intermediates.noindex/Runner.build/Release-iphoneos/Runner.build/Script-9740EEB61CF901F6004384FC.sh
♦ /Users/gudmhb/development/flutter/bin/flutter --verbose assemble --no-version-check --output=/Users/gudmhb/Projects/emote/build/ios/Release-iphoneos/ -dTargetPlatform=ios -dTargetFile=/lib/main.dart -dBuildMode=release -dIosArchs=arm64 -dSdkRoot=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS18.1.sdk -dSplitDebugInfo= -dTreeShakeIcons=true -dTrackWidgetCreation=false -dDartObfuscation=false -dAction=build -dFrontendServerStarterPath= --ExtraGenSnapshotOptions= --DartDefines=FLAVOR=production --ExtraFrontEndOptions= -dCodesignIdentity=AF77BA517530838410D3ABF23F3410148111EC01 release_ios_bundle_flutter_assets
[  +32 ms] executing: sysctl hw.optional.arm64
[  +17 ms] Exit code 1 from: sysctl hw.optional.arm64
[        ] sysctl: unknown oid 'hw.optional.arm64'
Artifact Instance of 'FontSubsetArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'PubDependencies' is not required, skipping update.
[  +38 ms] "flutter assemble" took 173ms.
[   +1 ms] Running 1 shutdown hook
[        ] Shutdown hooks complete
[   +1 ms] exiting with code 1
error: [   +5 ms] Error parsing assemble command: your generated configuration may be out of date. Try re-running 'flutter build ios' or the appropriate build command.
[   +2 ms]
#0      throwToolExit (package:flutter_tools/src/base/common.dart:10:3)
#1      AssembleCommand.runCommand (package:flutter_tools/src/commands/assemble.dart:308:7)
#2      FlutterCommand.verifyThenRunCommand (package:flutter_tools/src/runner/flutter_command.dart:1815:12)

#3      FlutterCommand.run. (package:flutter_tools/src/runner/flutter_command.dart:1450:27)

#4      AppContext.run. (package:flutter_tools/src/base/context.dart:153:19)

#5      CommandRunner.runCommand (package:args/command_runner.dart:212:13)

#6      FlutterCommandRunner.runCommand. (package:flutter_tools/src/runner/flutter_command_runner.dart:421:9)

#7      AppContext.run. (package:flutter_tools/src/base/context.dart:153:19)

#8      FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:364:5)

#9      run.. (package:flutter_tools/runner.dart:131:9)

#10     AppContext.run. (package:flutter_tools/src/base/context.dart:153:19)

#11     main (package:flutter_tools/executable.dart:94:3)

Failed to package /Users/gudmhb/Projects/emote.`
Logs

Logs

<!-- Paste your logs here -->

Flutter Doctor output

Doctor output

(base) gudmhb@Gumundurs-MacBook-Pro ios % flutter doctor -v
[✓] Flutter (Channel stable, 3.27.1, on macOS 14.6.1 23G93 darwin-x64, locale en-US)
    • Flutter version 3.27.1 on channel stable at /Users/development/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 17025dd882 (2 days ago), 2024-12-17 03:23:09 +0900
    • Engine revision cb4b5fff73
    • Dart version 3.6.0
    • DevTools version 2.40.2

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/gudmhb/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 16.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 16B40
    • CocoaPods version 1.16.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)

[✓] VS Code (version 1.92.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.94.0

[✓] VS Code (version 1.83.1)
    • VS Code at /Users/gudmhb/Downloads/Visual Studio Code.app/Contents
    • Flutter extension version 3.94.0

[✓] VS Code (version 1.84.0)
    • VS Code at /Users/gudmhb/Downloads/Visual Studio Code-2.app/Contents
    • Flutter extension version 3.94.0

[✓] Connected device (3 available)
    • iPhone 16 Pro (mobile) • 619559C6-F401-43AF-AC3A-33E676C25EF3 • ios            •
      com.apple.CoreSimulator.SimRuntime.iOS-18-1 (simulator)
    • macOS (desktop)        • macos                                • darwin-x64     • macOS 14.6.1 23G93
      darwin-x64
    • Chrome (web)           • chrome                               • web-javascript • Google Chrome 131.0.6778.140

[✓] Network resources
    • All expected network resources are available.

• No issues found!

@gummz
Check if this helps in your case or not.

Unfortunately not, as “which sysctl” returns the location on my machine.
fim., 19. des.. 2024 klukkan 07:45, darshankawar @.***(mailto:fim., 19. des.. 2024 klukkan 07:45, darshankawar < skrifaði:

@.***(https://github.com/gummz)
Check if this helps in your case or not.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

It seems you are hitting https://github.com/flutter/flutter/issues/134766, so please follow-up in it for further updates. You may share your findings in it for better tracking.
Closing this as a duplicate.


[SPARK-19762][ML][FOLLOWUP]Add necessary comments to L2Regularization.

What changes were proposed in this pull request?
MLlib LinearRegression/LogisticRegression/LinearSVC always standardize the data during training to improve the rate of convergence regardless of standardization is true or false. If standardization is false, we perform reverse standardization by penalizing each component differently to get effectively the same objective function when the training dataset is not standardized. We should keep these comments in the code to let developers understand how we handle it correctly.
How was this patch tested?
Existing tests, only adding some comments in code.

Test build #80839 has started for PR 18992 at commit 88b20dc.

@sethah @srowen Thanks for your great contributions for #17094. I wish you would not mind, I found these annotation section was missing. I think this is very important to let users/developers to understand how we handling standardization, so I add it back. Would you mind to have a look? Thanks.

Test build #80839 has finished for PR 18992 at commit 88b20dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80839/
Test PASSed.

Merged into master. Thanks.


Simplified Chinese Support

Add Simplified Chinese Support. I hope this file can be accepted

Coverage remained the same at 48.409% when pulling 16ac4a4d538544fd2ede6ac6644c3eb7b9fed6c2 on Meodinger:zh-cn into 4d8ea05b8c2648301c21fdc1a57c136041704a60 on LoveLiveSunshine:master.


Change neo4j password to test

We use test in databuilder sample loader
Summary of Changes
Include a summary of changes then remove this line
Tests
What tests did you add or modify and why? If no tests were added or modified, explain why. Remove this line
Documentation
What documentation did you add or modify and why? Add any relevant links then remove this line
CheckList
Make sure you have checked all steps below to ensure a timely review.

[ ] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"

In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

[ ] PR includes a summary of changes, including screenshots of any UI changes.
[ ] PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
[ ] In case of new functionality, my PR adds documentation that describes how to use it.

All the public python functions and the classes in the PR contain docstrings that explain what it does

[ ] PR passes all tests documented in the developer guide
[ ] I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message"

Codecov Report

Merging #168 into master will increase coverage by 19.85%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #168       +/-   ##
==========================================
+ Coverage   53.24%   73.1%   +19.85%
==========================================
  Files         108      22       -86
  Lines        2710     948     -1762
  Branches      277      79      -198
==========================================
- Hits         1443     693      -750
+ Misses       1238     229     -1009
+ Partials       29      26        -3

Impacted Files
Coverage Δ

...ication/static/js/components/common/Tabs/index.tsx

...ndsen_application/static/js/ducks/search/api/v0.ts

...ication/static/js/components/common/Flag/index.tsx

...ts/common/ResourceListItem/TableListItem/index.tsx

...sen_application/static/js/ducks/allTags/reducer.ts

...tatic/js/components/common/ScrollTracker/index.tsx

...pplication/static/js/ducks/popularTables/api/v0.ts

...js/components/TableDetail/WatermarkLabel/index.tsx

...sen_application/static/js/config/config-default.ts

...n/static/js/components/common/InfoButton/index.tsx

... and 76 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c54f92...6bc7025. Read the comment docs.

cc @jinhyukchang @jornh

Wow! Already merged ... so no real need to comment I guess.
Anyway - just a note I wondered about yesterday - it didn't really seem to care about what I put in this line:
https://github.com/lyft/amundsenfrontendlibrary/blob/3c59bccdaa099f5a280743fb922abbb47724aab7/docker-amundsen.yml#L49


[rv_dm,dv] Verify DTM idle behavior (jtag_dtm_idle_hint BLT)

This issue tracks the implementation of the jtag_dtm_idle_hint block-level test, which is specified in the testplan for rv_dm.

estimate 8

Triaged for rv_dm. Assigning to M2.5 with https://github.com/lowRISC/opentitan/labels/Priority%3AP2 because this is an open V2 test.

@rswarbrick  @jdonjdon need to figure out with impare whether this is part of the RV_DM testplan already (and hence addressed).

Implemented in 4143382e7ed203ad4726fc355fd0af063b4109eb (part of PR #20009) and merged last October.


Strange markdown convertion behavior

Hi,
Thanks for your work, very useful to me.
When using dataframe_image to produce a markdown conversion of my notebooks, I notice that part of the markdown result is ... html (paragraphs, sections, ...) !
I looked at the source code but I'm not familiar with the usage of nbconvert as a library.
All I can say is that a MarkdownExporter is used but before that a MarkdownPreprocessor is called.
If I remove it, my problem is solved.
I guess that Preprocessing of MarkDown cells is needed when rendering to latex/pdf, but I think it is unecessary when rendering to markdown.
A simple fix in convert() from _convert.py function would be (line 327 in master):
if "md" not in self.to:
    MarkdownPreprocessor().preprocess(self.nb, self.resources)

What do you think about it ?
If you wish, I can do a PR. I have never done that before but it's an opportunity to learn :)

Can you upload a notebook facing this problem? so we can test it to findout the problem.

Sorry for the delay, but here it is.
minimal_markdown.zip

@keufran , thanks for reporting this bug, it have been fix in the latest version.
Please try to install version 0.1.3 and see if the bug still exist.

It works, thank you !


Update absolute path example in route config guide

This is the slightly more verbose but clearer option that it seems like most of y'all preferred in https://github.com/reactjs/react-router/issues/3172.

@ryanflorence Is this what you had in mind as the "clearer" replacement for nested absolute paths?

I think this is clearer than what's currently on master.


onPress’s value always returns as false

Am I doing something wrong? Keep getting false on the onPress function.
Using the button with redux:
<RadioForm style = {{ alignSelf: 'flex-start', marginTop: 12 }}
            radio_props = {[{label: 'Show password', value: this.props.showPassword }]}
            buttonSize = { 12 }
            labelStyle = {{ fontSize: 13, color: '#535353', paddingLeft: 6}}
            buttonColor = { '#E52C51' }
            initial = { this.props.showPassword }
            onPress = { this.onShowPasswordToggle.bind(this) }
/>

onShowPasswordToggle(toggleState: boolean) {
    this.props.showPasswordToggeled(toggleState)
}

My mistake, I was trying to use it as CheckBob controller...


Docs and nulls

Added docs and fixed null order

Codecov Report

Merging #43 (7279101) into main (d9283f0) will increase coverage by 0.69%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #43      +/-   ##
==========================================
+ Coverage   89.08%   89.77%   +0.69%
==========================================
  Files           2        2
  Lines         174      176       +2
==========================================
+ Hits          155      158       +3
+ Misses         19       18       -1

Impacted Files
Coverage Δ

src/pydantic_avro/avro_to_pydantic.py
82.60% <100.00%> (ø)

src/pydantic_avro/base.py
97.61% <100.00%> (+1.27%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d9283f0...7279101. Read the comment docs.

@ojomio was this this ready? for me this looks fine :)

Sorry, was on vacations) Seems to be ready, will provide an update today

version 0.4.1 will include this

Thank you for support!


Debug/Fix CLI and OpenBCI Boards

[x] Fixes #40
[x] Add support for Ganglion+WiFi via CLI
[x] Add newline characters as needed to make input prompts more readable and organized for users

@JohnGriffiths @JadinTredup @m9h
Full Console Log using Input Prompt and Ganglion w/ BLED112 Dongle on Mac OS 10.15.7:
(eeg-nb-main) Richard-Waltmans-iMac-4:eeg-notebooks Richard$ eegnb runexp -ip
run command line prompt script
Welcome to NeurotechX EEG Notebooks.
Please enter the integer value corresponding to your EEG device:
[0] None
[1] Muse2016
[2] Muse2
[3] MuseS
[4] OpenBCI Ganglion
[5] OpenBCI Cyton
[6] OpenBCI Cyton + Daisy
[7] G.Tec Unicorn
[8] BrainBit
[9] Notion 1
[10] Notion 2
 [11] Synthetic

Enter Board Selection: 4
Selected board OpenBCI Ganglion

Please select your connection method:
[0] usb dongle
[1] wifi shield

Enter connection method: 0

Ganglion MAC Address (Press Enter to Autoscan):

Please select which experiment you would like to run:
[0] visual n170
[1] visual p300
[2] ssvep

Enter Experiment Selection: 0
Selected experiment visual-N170

Now, enter the duration of the recording (in seconds).

Enter duration: 12

Next, enter the ID# of the subject you are recording data from.

Enter subject ID#: 1

Next, enter the session number you are recording for.

Enter session #: 1

Getting a list of available serial ports...
[0] /dev/cu.MyLovesAirPods-Wireless - n/a
[1] /dev/cu.UEBOOM2-LWACP - n/a
[2] /dev/cu.Bluetooth-Incoming-Port - n/a
[3] /dev/cu.usbserial-DM00D7TW - FT231X USB UART - FT231X USB UART
[4] /dev/cu.usbmodem11 - Low Energy Dongle - CDC data
Select Port(number): 4
[2020-12-02 23:20:09.628] [brainflow_logger] [info] incomming json: {
    "ip_address": "",
    "ip_port": 0,
    "ip_protocol": 0,
    "mac_address": "",
    "other_info": "",
    "serial_number": "",
    "serial_port": "/dev/cu.usbmodem11",
    "timeout": 0
}
[2020-12-02 23:20:09.629] [brainflow_logger] [info] use 15 as a timeout for device discovery and for callbacks
[2020-12-02 23:20:09.629] [brainflow_logger] [info] mac address is not specified, try to find ganglion without it

EEG device successfully connected!
Press [ENTER] when ready to begin...
21.4626         WARNING         User requested fullscreen with size [1600  900], but screen is actually [2560, 1440]. Using actual size
23.0569         WARNING         User requested fullscreen with size [1600  900], but screen is actually [2560, 1440]. Using actual size
[2020-12-02 23:20:20.491] [brainflow_logger] [info] use command b to start streaming

Ganglion+WiFi on Mac OS 10.15.7:
(eeg-nb-main) Richard-Waltmans-iMac-4:eeg-notebooks Richard$ eegnb runexp -ip
run command line prompt script
Welcome to NeurotechX EEG Notebooks.
Please enter the integer value corresponding to your EEG device:
[0] None
[1] Muse2016
[2] Muse2
[3] MuseS
[4] OpenBCI Ganglion
[5] OpenBCI Cyton
[6] OpenBCI Cyton + Daisy
[7] G.Tec Unicorn
[8] BrainBit
[9] Notion 1
[10] Notion 2
 [11] Synthetic

Enter Board Selection: 4
Selected board OpenBCI Ganglion

Please select your connection method:
[0] usb dongle
[1] wifi shield

Enter connection method: 1

Enter Ganglion+WiFi IP Address: <IP_ADDRESS>

Please select which experiment you would like to run:
[0] visual n170
[1] visual p300
[2] ssvep

Enter Experiment Selection: 0
Selected experiment visual-N170

Now, enter the duration of the recording (in seconds).

Enter duration: 12

Next, enter the ID# of the subject you are recording data from.

Enter subject ID#: 1

Next, enter the session number you are recording for.

Enter session #: 1
[2020-12-03 01:00:02.078] [brainflow_logger] [info] incomming json: {
    "ip_address": "<IP_ADDRESS>",
    "ip_port": 6677,
    "ip_protocol": 0,
    "mac_address": "",
    "other_info": "",
    "serial_number": "",
    "serial_port": "",
    "timeout": 0
}
[2020-12-03 01:00:02.078] [brainflow_logger] [info] use 10 as http timeout
[2020-12-03 01:00:02.078] [brainflow_logger] [info] local ip addr is <IP_ADDRESS>
[2020-12-03 01:00:04.115] [brainflow_logger] [info] configuration string {"delimiter":true,"ip":"<IP_ADDRESS>","latency":10000,"output":"raw","port":6677}
[2020-12-03 01:00:06.175] [brainflow_logger] [info] command string {"command":"~4"}
[2020-12-03 01:00:08.291] [brainflow_logger] [warning] If you change gain you may need to rescale EXG data

EEG device successfully connected!
Press [ENTER] when ready to begin...
28.7602         WARNING         User requested fullscreen with size [1600  900], but screen is actually [2560, 1440]. Using actual size
29.6554         WARNING         User requested fullscreen with size [1600  900], but screen is actually [2560, 1440]. Using actual size
(eeg-nb-main) Richard-Waltmans-iMac-4:eeg-notebooks Richard$


schedule s3fs (Access Denied)

Hi Team,
I currently use s3fs (for S3 Storage instead of Local storage) to schedule data pipelines everyday . For first time, after initialize a s3fs.S3FileSystem() instance, all pipelines successfully ran, but next days, they are all failed (the old files still successfully run). I found that all functions (ls(), info(),...) have refresh parameters. However, I still get Access Denied for new files, it seems consistent. Could you do any things to fix it?
Regards
Giang

What version are you running? This may be fixed in fsspec and s3fs master.

Also, what method do you use to auth, any special arguments to S3FileSystem? If you are using roles and STS, you might find this interesting reading: https://dev.to/li_chastina/auto-refresh-aws-tokens-using-iam-role-and-boto3-2cjf

Hi Martin,
My version is 0.3.4, I used default initialization with no parameters.
fs= s3fs.S3FileSystem()

Sorry, I got a little lost there - could you post the code, please?

Hi Martin,
I used functions ls and open (sometimes exists) many times in one process in day. So I cannot track back. But I founded a solution it is clear_instance_cache(), it solved problem. However, for now, a documentation hasn't updated yet.

Additionally, an error is like this

But successful when it started again.


Mention jupyter/docker stacks as an alternative to rocker/binder

rocker/binder contains some Jupyter things, but the rocker project as a whole is not focused on Jupyter. This links them out to the jupyter/docker-stacks project, which is more Jupyter focused to serve users who are looking for more Jupyter focused container images. I think a lot of communities / users use a combination of Python (from various sources) and R (from the rocker project) already anyway.
I link to the rocker project from jupyter/docker-stacks since last month (https://github.com/jupyter/docker-stacks/pull/2065), and one of the maintainers asked if I could provide a link back as well.

Thanks, but the link to Jupyter Docker Stacks has been present on a dedicated page for a few years.
https://github.com/rocker-org/website/blob/f8983df9b9d1ac74507907863fda512360c14856/images/external.md?plain=1#L10-L19
Do we need to link to it from the binder page as well?

@eitsupi ah, that's great it's already there. However, I do think it's also useful to link to it specifically from the binder page, as that's where someone most likely is going to be looking for things related to mixing R & Jupyter. Perhaps we can just link out to the external section (i.e. this link https://rocker-project.org/images/external.html#jupyter-docker-stacks) instead of straight to jupyter/docker-stacks?

Perhaps we can just link out to the external section (i.e. this link rocker-project.org/images/external.html#jupyter-docker-stacks) instead of straight to jupyter/docker-stacks?

That makes sense.

@eitsupi done!


Feature: DX12 support for dumping draw call resources

CI gfxreconstruct build queued with queue ID 144468.

CI gfxreconstruct build # 3812 running.

CI gfxreconstruct build # 3812 passed.

CI gfxreconstruct build queued with queue ID 150073.

CI gfxreconstruct build # 3867 running.

CI gfxreconstruct build # 3867 passed.

CI gfxreconstruct build # 3885 running.

CI gfxreconstruct build queued with queue ID 161891.

CI gfxreconstruct build # 3959 running.

CI gfxreconstruct build # 3959 passed.

CI gfxreconstruct build # 4096 running.

CI gfxreconstruct build # 4096 passed.

CI gfxreconstruct build queued with queue ID 181277.

CI gfxreconstruct build # 4123 running.

CI gfxreconstruct build # 4123 failed.

CI gfxreconstruct build queued with queue ID 182591.

CI gfxreconstruct build # 4136 running.

CI gfxreconstruct build # 4136 passed.

CI gfxreconstruct build queued with queue ID 182629.

CI gfxreconstruct build # 4137 running.

CI gfxreconstruct build # 4137 passed.

CI gfxreconstruct build queued with queue ID 184433.

CI gfxreconstruct build # 4148 running.

CI gfxreconstruct build # 4148 passed.

CI gfxreconstruct build queued with queue ID 189033.

CI gfxreconstruct build # 4221 running.

CI gfxreconstruct build # 4221 passed.


Change to 'new' constructor pattern

Remains backwards-compat with function-call instantiation.
See https://github.com/hypercore-protocol/hyperbee/pull/19 for history

Do we want to do this with hypercore too?

Do we want to do this with hypercore too?

Looks good to me given green CI, will let @andrewosh get a chance to look at it also

Looks good to me given green CI, will let @andrewosh get a chance to look at it also

Now that's some JS. Looks good to me

Now that's some JS. Looks good to me

Published 10.20.0

Published 10.20.0

Is the backwards compat going to be removed in the next major release?

Is the backwards compat going to be removed in the next major release?

@RangerMauve i don't think so, but at some point, just because classes are a bit easier :)

@RangerMauve i don't think so, but at some point, just because classes are a bit easier :)


deployment for us-west-2?

one of the more popular regions, need to have one for it as well.

You implemented this in your PR, thank you! Closing this issue.


Update docker integration put to use artifact manager

Currently the GET /blobs reads from the artifact manager but there is no programatic way to write data there.
This is a regression on main

Yes to first question and there will be Erwin's changes as he shown today in demo is to pull from dockerhub if not there locally and then store to local artifact_manager. Second is good question but right now we do not full e2e working mechanism for that but the end thought is to have handle_put_blobs will go through artifact_manager to store the blobs at same place based on digest. Also handle_put_blobs does appending of data, so we have to see how it will play through our current implementation of pu_artifact
Originally posted by @MitaliBo in https://github.com/pyrsia/pyrsia/pull/161#discussion_r774762169

https://github.com/pyrsia/pyrsia/pull/175
Did the opposite.
We can now write manifest to the artifact manager but there is no way to extract this data.

Our "demo" is broken since we can no longer push and pull 😱

Seems like issue(s) exist?
#195 covers half

@prince-chrismc We can close this now as we have mechanism now to pull manifest and blobs from local pyrsia storage and if we don't have it then we pull them from dockerhub and store in pyrsia.


(PUP-4678) Allow yumrepo priority below the default

The default priority is 99 so limiting it with puppet to
99 too made it impossible to add a repository with a lower
priority to protect system packages. This change removes
this limitation.

Can one of the admins verify this patch?

CLA signed by all contributors.

You are right, @jhelwig . Fixed. Let me know if what I did is ok, then I will squash my commits.

Hi @gdubicki, with the 5.5.x branch being cut, please retarget that branch.  Thanks!


Properly handle stream selection and selected_properties, including both RECORD and SCHEMA messages

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/7
Originally created by @aaronsteers on 2020-12-12 03:13:16

This would add proper stream selection and property selection via the input catalog (when provided).
Approach
Following from the discussion here, the selection filtering should occur at four steps in the process:

Selecting or deselecting entire streams.
As an input to get_records() - so tap developers can avoid collecting data they don't need.

This is available today in some manner, if the developer interrogates Tap.input_catalog (not a trivial effort).

As a property filter on the standard (base class) transformation from raw data into the RECORD message.

Failsafe for excessive fields being returned from get_records().

As a property filter on the standard (base class) generation of SCHEMA messages.

Many taps don't perform this filtering today, which results in downstream columns being unnecessarily (and confusingly) created in the target's destination tables.

View 5 previous comments from the original issue on GitLab


[kjobctl] Split samples.

What type of PR is this?
/kind cleanup

What this PR does / why we need it:
Split kjobctl samples.
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
It’s very difficult to locate the kjobctl configuration for each mode. Splitting the configurations into separate files would simplify navigation and make it easier to identify the minimum requirements for each mode.
Does this PR introduce a user-facing change?

NONE

/cc @trasc

/lgtm


Doesn't support env var --> SYNTHTOOL_GOOGLEAPIS (local googleapis)

Is your feature request related to a problem? Please describe.
It doesn't support $SYNTHTOOL_GOOGLEAPIS (env var that points to local googleapis/googleapis).  Other languages supports $SYNTHTOOL_GOOGLEAPIS, and I thnk it might be good idea to add this as option and keep it consistent throughout all languages.
Describe the solution you'd like
Add an option to read $SYNTHTOOL_GOOGLEAPIS (points to local googleapis) if that env variable exists.

Most of our generation tooling predates synthtool, and assumes googleapis is directly under the root directory - cloning it if necessary. Changing that might be significant work - mostly in terms of tracking down everywhere I expect to find googleapis there. Is there a concrete use case that would make that work worthwhile?

The use case is to be able to edit .proto files locally and generate libraries from them in more than 1 language. I generate libraries in multiple languages from local .proto files with changes, so there should be 1 common location on my computer where googleapis lives.

Okay, I'll take a look. No promises just yet though.

Okay, I've created #3340 and tested it with a copy of googleapis with a change to KMS.
It's likely this will be the only synthtool environment variable we'll implement for now:

SYNTHTOOL_ARTMAN_VERSION: We don't use Artman at all
SYNTHTOOL_GENERATOR: It's likely to be Chris (or myself) working on generator changes anyway. Once we've moved over to the microgenerator wholeheartedly, we might revisit this if it becomes useful.


GitHub Copilot Chat crashing whole IDE with OOM

Copilot Chat Extension Version: v0.20.0
VS Code Version:
Version: 1.93.0 (system setup)
Commit: 4849ca9bdf9666755eb463db297b69e5385090e3
Date: 2024-09-04T13:02:38.431Z
Electron: 30.4.0
ElectronBuildId: 10073054
Chromium: 124.0.6367.243
Node.js: 20.15.1
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22631

Steps to Reproduce:

Open VS Code
Open Github Copilot Chat - Select some code and ask a question.

Interestingly the crash happens only when a folder is opened. If there is no folder or workspace opened, the chat works fine

@TylerLeonhardt @aeschli, I have the dmp file, let me know how should I send it

The dump file suggests that Electron is crashing on a callback involving processing a URL rather than there being an OOM error.

When there is no folder or workspace opened, does the chat output any URLs or file paths?
Does the crash occur when a single file is opened (e.g. by running code some_file.js)?

The dump file suggests that Electron is crashing on a callback involving processing a URL rather than there being an OOM error.

When there is no folder or workspace opened, does the chat output any URLs or file paths?
Does the crash occur when a single file is opened (e.g. by running code some_file.js)?

When there is no file or workspace opened, the chat works fine
The crash didn't occur when a single file is opened


Renaming folder breaks file selection

Create a new project
Create a file "file.js" at the top level
Create a folder "folder" at the top level
Create a file "file-in-folder.js" inside "folder"
Select the top-level "file.js"
Rename "folder" to something else
Try to select "file-in-folder.js"

Result: you can't - it goes into rename state. Any files in the folder end up in a confused state. Collapsing/expanding the folder doesn't fix it.

This seems important to fix - it pretty much breaks the file tree until you switch projects.

@madanbn Do you think we can mark this for 1.2 and get a fix in?

@njx I can't reproduce it on my Mac 10.9.

I'm not seeing it either in latest master on 10.10.2 .

Let me pull master - this was in 10.1 release.

I can reproduce it in master with a slightly altered set of steps. Updated above.

Yes, now I can reproduce it with the updated steps even in release 44.

I put up a possible fix in #10541, but there are other issues that someone more familiar with the file tree code should probably address.

There's a separate bug with the view panes if you follow these steps and then reselect "file-in-folder.js" again (see https://github.com/adobe/brackets/pull/10541#issuecomment-73821322), but the tree bug is fixed -- closing.


Adding in impact parameter and sky velocity

See whether we can do this by passing the tt array, reshaped to 3 dimension, with the first dimension equal to one.  Then, if we pass a tt array with the first dimension equal to three, the code will also compute b and v_{sky}.

This has been implemented in TransitParameters. (See src/transits/parameters.jl)


Deprecated feature in RabbitMQ

Getting the following error when trying to deploy the easytrade

By adding the following lines to the rabbitmq.yaml and restarting the pod fixes the issue
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-config
data:
  rabbitmq.conf: |
    # RabbitMQ configuration
    management.load_definitions = /etc/rabbitmq/definitions.json

    # Permit deprecated feature
    deprecated_features.permit.management_metrics_collection = true

Hi,
I've checked this and the deprecation notice seems to be shown anyway even with the manifest change you suggested or when I change the config in the image directly. I don't think we're using this feature directly so this shouldn't be an issue but I'll push the change with the updated config.


Allow survey banner to be shown with cookie banner

We were previously preventing the survey banner from being shown when any other banners were present, including the cookie banner. Now that the cookie banner is persistent, we want to allow both to show at the same time.

https://trello.com/c/cRBewa76

https://trello.com/c/cRBewa76