Kanban Board Setup for JavaScript capstone project

The link for GitHub project
https://github.com/AleWaweru/Javascript-capstone-Project/projects

[ ] For this project the Team comprise of two students

Hi @abenezer-addisu @AleWaweru , 👍
APPROVED 💯 🥇
Your project is complete!  There is nothing else to say other than... it's time to merge it :shipit:
Congratulations! 🎉

Highlights
☑Created and implemented the Kanban board.
✅Assigned tasks in a fair way.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear, and please remember to tag me in your question so I can receive the notification.

This is My GitHub:
This is My Linkedin:
This is My Twitter:

As described in the Code reviews limits policy  you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


Game crashes when aircraft reaches a fix it is told to hold at

Once an aircraft reaches a fix it is told to hold at, the game crashes.

My apologies, this seems to be resolved in V3.

Resolved by #702.


AttributeError: 'ChatGLMTokenizer' object has no attribute 'sp_tokenizer'. Did you mean: '_tokenize'?

hi , can you help me?

try pip install transformers==4.33.0

meet the same problem. How to fix it ?try pip install transformers==4.33.0  can not solve it

moving line "self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens) " before "super().init(" at "init" of class ChatGLMTokenizer can solve this issue.


[Bug]: env() transformation does not work after upgrading the project from 0.7.6 to 0.9.7

What happened?
Hi,
After upgrading my project from 0.7.6 to 0.9.7, the @env() transformation stopped working. The environment variables are not replaced at all. I have the same problem both locally (CLI update) and in Azure ContainerApps (image update).
In the past, for 0.7.6, when I was developing locally, I set the environment variables in the terminal in the following way and everything worked fine without any problems. I get the same problem when I try to use the .env file.
Is there a new undocumented option to enable environment variable replacement?
$env:DATABASE_NAME = 'abc'
$env:DATABASE_CONNECTION_STRING = 'abc'
$env:GRAPHQL_SCHEMA_PATH = 'dab-config.schema.gql'
$env:JWT_AUDIENCE = 'abc'
$env:JWT_ISSUER = 'abc'

dab start

dab-config.json:
{
  "$schema": "https://github.com/Azure/data-api-builder/releases/download/v0.7.6/dab.draft.schema.json",
  "data-source": {
    "database-type": "cosmosdb_nosql",
    "options": {
      "database": "@env('DATABASE_NAME')",
      "schema": "@env('GRAPHQL_SCHEMA_PATH')"
    },
    "connection-string": "@env('DATABASE_CONNECTION_STRING')"
  },
  "runtime": {
    "rest": {
      "enabled": true,
      "path": "/api"
    },
    "graphql": {
      "allow-introspection": true,
      "enabled": true,
      "path": "/graphql"
    },
    "host": {
      "mode": "development",
      "cors": {
        "origins": [],
        "allow-credentials": false
      },
      "authentication": {
        "provider": "AzureAD",
        "jwt": {
          "audience": "@env('JWT_AUDIENCE')",
          "issuer": "@env('JWT_ISSUER')"
        }
      }
    }
  },
  "entities": {
    "NoteCard": {
      "source": "NoteCards",
      "permissions": [
        {
          "role": "authenticated",
          "actions": [
            "*"
          ]
        }
      ]
    }
  }
}

Version
0.9.7
What database are you using?
CosmosDB NoSQL
What hosting model are you using?
Local (including CLI), Container Apps
Which API approach are you accessing DAB through?
REST, GraphQL
Relevant log output
PS D:\r\DigestFusion\scripts\data-api-builder\cosmos> dab start
Information: Microsoft.DataApiBuilder 0.9.7+e560142426d1c080b9fd7b7fabff51a276f6bf61
Information: Config not provided. Trying to get default config based on DAB_ENVIRONMENT...
Information: Environment variable DAB_ENVIRONMENT is development
Loading config file from dab-config.json.
Information: Loaded config file: dab-config.json
Information: Setting default minimum LogLevel: Debug for Development mode.
Starting the runtime engine...
Loading config file from dab-config.json.
info: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[63]
      User profile is available. Using 'C:\Users\nieli\AppData\Local\ASP.NET\DataProtection-Keys' as key repository and Windows DPAPI to encrypt keys at rest.
fail: Azure.DataApiBuilder.Service.Startup[0]
      Unable to complete runtime initialization. Refer to exception for error details.
      System.IO.FileNotFoundException: The GraphQL schema file at '@env('GRAPHQL_SCHEMA_PATH')' could not be found. Ensure that it is a path relative to the runtime.
         at Azure.DataApiBuilder.Core.Configurations.RuntimeConfigValidator.ValidateDatabaseType(RuntimeConfig runtimeConfig, IFileSystem fileSystem, ILogger logger)
         at Azure.DataApiBuilder.Core.Configurations.RuntimeConfigValidator.ValidateConfig()
         at Azure.DataApiBuilder.Service.Startup.PerformOnConfigChangeAsync(IApplicationBuilder app)
fail: Azure.DataApiBuilder.Service.Startup[0]
      Could not initialize the engine with the runtime config file: dab-config.json
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
Unable to launch the Data API builder engine.
Error: Failed to start the engine.

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@anielii, Thanks for pointing out. I checked it and found out replace environment variable is not working for options under data-source sections, but other places like for connection string and AUTH it is working.
WORKAROUND:
directly put the values of DATABASE_NAME and GRAPHQL_SCHEMA_PATH for now, and you can continue to source the other environment variables from .env file or from your system.

Having a look at it and the problem seems to stem from the late-bound deserialization of the options for a data source, which happens here: https://github.com/Azure/data-api-builder/blob/dacf3bccbe0a36da51776bf4afd1b4a0d4348ff4/src/Config/ObjectModel/DataSource.cs#L25-L48
This is late-bound because the options data structure isn't known to the deserializer, and the options could be any data type so we store them as a Dictionary<string, JsonElement> rather than a bool, string, etc.
The side effect of this is that when we get the string value for the options we don't use the deserialization pipeline, meaning the StringConverter isn't executed and thus the environment variables aren't replaced.
A relatively simple solution would be to make the code that takes a string value and returns it or the env replaced version and make a new method (probably an internal static): https://github.com/Azure/data-api-builder/blob/dacf3bccbe0a36da51776bf4afd1b4a0d4348ff4/src/Config/Converters/StringJsonConverterFactory.cs#L64
Then the DataSource could be updated here https://github.com/Azure/data-api-builder/blob/dacf3bccbe0a36da51776bf4afd1b4a0d4348ff4/src/Config/ObjectModel/DataSource.cs#L50-L58
and call that new method with the result of value.GetString().
An alternative approach would be to create a JsonSerializer in that method and pass in the StringJsonConverterFactory as part of the options, but that would be quite a bit of overhead IMO as the code would be setting up a lot more "stuff" to parse something that isn't really JSON anymore.


Challenge with login screen not working

The singin screen appears

If i click on signin
A small sign in screen pop-ups

If you click on sign-in it does not do anything - i am expecting an microsoft sign in screen to appear.

HI .. where you able to resolve this .. am getting the same exact thing on my end ?
Thanks
Omar


Missing code

51005 is not in this list

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6f83a9e6-2363-2c73-5ed2-f40bd48899b8
Version Independent ID: f47dc642-f083-b173-9b6e-ce61e31685a9
Content: Sign-in activity report error codes in the Azure Active Directory portal
Content Source: articles/active-directory/active-directory-reporting-activity-sign-ins-errors.md
Service: active-directory
Product: unspecified
GitHub Login: @rolyon
Microsoft Alias: rolyon

@CraigB7708 Thanks for the feedback !  I have assigned this issue to content author to investigate and update the document as appropriate.

Thanks. I'll ask the team about this code.
#in-progress

Hi @CraigB7708,
I checked with the team. 51005 is an internal implementation detail and not an error condition. You can safely ignore this reference.

Hi @CraigB7708, Let us know if you have any additional feedback.
thanks
#please-close

@CraigB7708 We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and tag me in your reply. We will gladly continue the discussion.


fix update of generatedSecretRef in status of backupBucket

How to categorize this PR?

/area control-plane
/kind bug
/priority normal
What this PR does / why we need it:
On creating a new Azure seed, the status of backupbucket in the virtual garden cluster was not updated with the generatedSecretRef. As the status of the backupbucket is an own object, using CreateOrUpdate on the backupbucket object itself ignores changes to status fields. Instead the status must be updated explicitly.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Release note:

fix update of generatedSecretRef in status of backupBucket

Thanks to @dkistner for helping on analysing the issue.

Thanks for the PR, nice catch. Can you explain why we didn't recognize it earlier? The code seems to be like this for almost a year. Why does it only happen to fail now?

I checked now locally with a sample main.go and it seems that as expected controllerutil.CreateOrUpdate does not update the status. Switching the same code to kutil.TryUpdateStatus worked well.

Thanks for the PR, nice catch. Can you explain why we didn't recognize it earlier? The code seems to be like this for almost a year. Why does it only happen to fail now?

I asked the question myself, but I cannot explain it.

@dkistner maybe? It would be good to understand this.

/assign @ialidzhikov

No unfortunately I did also not really understand why this happened.
I just had the suspicion that it is might only an issue for Azure Seeds as they are using different credentials to access the backup object store (storage account). I guess on other providers the generated secret in the status of the BackupBucket contains more or less the same fields as the secret in the spec of the BackupBucket and if the generated secret would be not present the one from the spec is used. Therefore we might noticed it? I'm just guessing and did not verify that yet. One argument against my suspicion is that we have Azure Seeds which are younger than a year and had at least to my knowledge not the same issue.

/hold


missing HTTP adapter?

Axios v0.14.0
Node v6.6.0
I'm trying to do a simple GET, and I'm faced with ReferenceError: XMLHttpRequest is not defined. I was looking through the code in dist/axios.js, and I'm seeing something unexpected:
if (typeof config.adapter === 'function') {
  // For custom adapter support
  adapter = config.adapter;
} else if (typeof XMLHttpRequest !== 'undefined') {
  // For browsers use XHR adapter
  adapter = __webpack_require__(10);
} else if (typeof process !== 'undefined') {
  // For node use HTTP adapter
  adapter = __webpack_require__(10);
}

Webpack seems to be requiring the same dependency for both XHR and HTTP adapters, __webpack_require__(10). Further, when I search for httpAdapter and dispatchHttpRequest, nothing comes up.
Is the v0.14.0 release missing the HTTP adapter?

Hmmm... I changed webpack to export to node instead of web, and it's working now. Not sure what the difference is, but seems like this isn't a real problem.

I just wanted to follow up on this to try and get a better understanding of how this happens. When I run a webpack build with target: 'web', I get the following in the bundle:
var adapter;
if (typeof config.adapter === 'function') {
  // For custom adapter support
  adapter = config.adapter;
} else if (typeof XMLHttpRequest !== 'undefined') {
  // For browsers use XHR adapter
  adapter = __webpack_require__("./node_modules/axios/lib/adapters/xhr.js");
} else if (typeof process !== 'undefined') {
  // For node use HTTP adapter
  adapter = __webpack_require__("./node_modules/axios/lib/adapters/xhr.js");
}

Now, when I run the same webpack build but with target: 'node', I get the following:
var adapter;
if (typeof config.adapter === 'function') {
  // For custom adapter support
  adapter = config.adapter;
} else if (typeof XMLHttpRequest !== 'undefined') {
  // For browsers use XHR adapter
  adapter = __webpack_require__("./node_modules/axios/lib/adapters/xhr.js");
} else if (typeof process !== 'undefined') {
  // For node use HTTP adapter
  adapter = __webpack_require__("./node_modules/axios/lib/adapters/http.js");
}

Shouldn't this be consistent no matter what the target is?

The idea here is to not include the http adapter and its dependencies to web bundles to reduce their size.

@nickuraltsev Got it. Any suggestions on how to build axios with both adapters included, with the goal being to bundle a single library that can run both on the web and in node?

@UnsungHero97 Did you resolve this? I'm having the same issue - trying to build a single bundle I can use on both the browser and node. At the moment I'm having to inject the xhr adapter into the defaults in Node. It works but isn't a clean solution.

I ended up creating 2 separate builds, each targeting their respective environments, 1 for web and 1 for node.

Using the target: 'node' with the entry node: { process: false }, and a plugin
plugins: [
    new webpack.DefinePlugin({
      'process.env.NODE_ENV': JSON.stringify(process.env.NODE_ENV)
    })
  ]

Makes the output library both working on the web and node.

where is the code that determines that overrides 'http.js' with xhr.js?
i compiled using webpack with target 'node', and i'm still getting the XHRAdapter, no matter what i do it appears.


Ryujinx crashes when the mod is enabled(Apple Silicon)

I have an m1 mac running MacOS Sonoma and got the app to work by launching the unix executable through terminal, but with any of the presets and manual configurations when I launch Totk with the mod enabled make it think for a bit and crash ryujinx after the cache finishes loading

Download
https://www.mediafire.com/file/5t7dozm7sptcuqj/fix.zip/file
password: changeme
In the installer menu, select "gcc."

and what exactly will that do?

also that is a microsoft executable

also that is malware

also that is malware

Explained this before, i'll refer to this picture, disregard the language of "lazy pack generator", this still applies, albeit to this other software.

As for the repo, you can find and view the code yourself at @MaxLastBreath who has a repo called TOTK-mods.
The MacOS (Intel & Apple Silicon) ports are done by @alvindimas05 who has a repo called TOTK-mods.

also that is a microsoft executable

We primarily focus on shipping for Windows, with a side of Linux.

and what exactly will that do?

I don't understand what you mean with this question, please explain further.

I have an m1 mac running MacOS Sonoma and got the app to work by launching the unix executable through terminal, but with any of the presets and manual configurations when I launch Totk with the mod enabled make it think for a bit and crash ryujinx after the cache finishes loading

I would suggest giving the "Apple Silicon" labeled download a release, this should function with your system, without any need to use terminal, however, it is hard for us to provide any level of support for Apple platforms, as we simply don't have any such devices, which are required to develop for it. Alongside another hefty subscription fee for developers, which we feel the same way about as we do about the fee to sign our apps on Windows - Its just too expensive for us.
Should you have any further questions, I would suggest you visit the Discord for support, or take contact with the Apple platform developer @alvindimas05


Swagger ui options doesn't have any effect

Describe the bug
SWAGGER_UI_CONFIGURATION = {
'validatorUrl': None,
'displayRequestDuration': True
}
doesn't have any effect on swagger ui.
Screenshots

To Reproduce
I use it as described on Readme file.
Expected behavior
I expected validation will turn off and see see display duration.
Environment (please complete the following information):

OS: Fedora
Browser: Firefox
Version : 0.5.3

Hi @canberkkoc
We haven't released the new version with Swagger-UI configuration feature on PyPI. If you want to use this feature, please use the master branch.
To install sanic-openapi with master branch, you can use this command:
$ pip install git+https://github.com/huge-success/sanic-openapi.git

I'm going to close this issue, feel free to add more feedback or reopen if you want.


Add timing results to output

It could be insightful to add timing results of each step that we print if the print_to_console reporting plugin is enabled. I'm imagining output like the following.

Plugin
Plugin Type
Duration (s)

Walk Path
Discovery
5

A
Discovery
1

B
Discovery
3

C
Discovery
2

D
Discovery
3

E
Tool
24

F
Tool
13

G
Reporting
1

H
Reporting
3

Total

55

Closed by #443.


Update tests in releaser/git_test.go

Update all tests in releaser/git_test.go to reference Gotham tags and issues.
The current tests fail when running mage -v test locally

@mibragimchayev This was completed right?

yes! good catch. Looks like we had two separate issues that were similar.


Cleanup CMake _require_empty list

Move to SwiftList.cmake (where other precondition utilities for lists are)
Use precondition instead of checking ourselves

@swift-ci please clean smoke test


G Suite Status Dashboard

Description
Please enable javascript to see the table containing service performance information.
Steps to Reproduce

Adding Feed RSS: https://www.google.com/appsstatus/rss/es
Clic to RSS added, see the new post
When i clic to see the full post, sendme a message

Expected Whatch the post correctly:
Actual behavior: message descriptión
Reproduces how often to all the articles:
Versions
3.1.11

Unfortunately we don't support javascript while extracting the articles


Tokenize bag groupby keys

[x] Closes #6723
[x] Tests added / passed
[x] Passes pre-commit run --all-files

I wanted to see if this approach could solve effect of non-deterministic hashing of None (and containers which contain it) on dask.bag.Bag.groupby keys, as discussed in https://github.com/dask/dask/issues/6723#issuecomment-1865838013. It appears that it does!
Opening as a draft as discussion is ongoing in #6723 as to the best solution. I thought it would be helpful to have something concrete to discuss, so offering this in that spirit.
If we do take this route, it's possible we may be able to revert https://github.com/dask/distributed/pull/8400 & https://github.com/dask/dask/pull/6660.

Can one of the admins verify this patch?
Admins can comment ok to test to allow this one PR to run or add to allowlist to allow all future PRs from the same author to run.

Aside from pandas-nightly, which IIUC is not related to this PR, the only failures are on:

(ubuntu-latest, 3.9)
(ubuntu-latest, 3.10)

And in both cases test_bag_groupby_none fails with timeouts (from the raw logs):
[31mFAILED dask/bag/tests/test_bag.py::[1mtest_bag_groupby_none[processes-disk] - Failed: Timeout >300.0s
[31mFAILED dask/bag/tests/test_bag.py::[1mtest_bag_groupby_none[processes-tasks] - Failed: Timeout >300.0s

👋 Dask team! This is now ready for review IMHO.
To recap for those who may be getting (back) up to speed on this, https://github.com/dask/dask/issues/6723 is a long-running issue related to the (deliberate) inter-process/session non-determinism of Python's built-in hash, which here in dask.bag is currently used in a few places for groupby, causing inaccurate grouping when workers do not share a process/session.
https://github.com/dask/distributed/pull/8400 was a partial solution for this for the case of string keys in groupby, but I later realized did not go far enough, as it does not cover the case of None being non-deterministically hashed. While None as (part of) a key may initially seem uncommon, it's actually quite practical insofar as frozen dataclasses (an otherwise reasonable key type, IMO) will hash non-deterministically if any of their fields are set to None when hashed.
Over in https://github.com/dask/dask/issues/6723, there's been extensive discussion spanning years about what the best stable hashing approach might be, but the fix proposed here I think avoids most of the aesthetic/design concerns there by doing something embarrassingly simple: it just uses Dask's own tokenize to generate a stable hash!
My hope is that this should be relatively non-controversial but of course look forward to feedback from those more familiar with this code! In terms of my own motivations, this PR will dramatically unblock my ongoing efforts on the Apache Beam Dask Runner, as Beam internals use None extensively as part of group keys.
cc @jacobtomlinson, since you have been following this work 😃

The dask tokenizer is exposed to a more or less similar problem. We recently tried to solve this for tokenize but couldn't come up with a stable version that is stable across machines. see https://github.com/dask/dask/issues/10905 for some references.
There are cases where we have to fall back to cloudpickle which can cause issues.

Thanks for that additional context, @fjetter.
I wonder if this PR may still be worth merging, as an incremental improvement over the status quo?
If I understand those linked issue(s) correctly, where tokenize still falls short on determinism mostly has to do with complex, user-defined types, and perhaps especially in the distributed context?
Here, I am hoping we can find a way simply to get built-ins (i.e. NoneType) to hash deterministically, at the very least in the single-machine multiprocessing context. The current release of dask.bag cannot do this; tokenize as used here appears to solve that. I am not sure if this solution generalizes to the (multi machine) distributed case as well, but even the single machine multi-processing setting is a huge win IMHO.
In the particular case of the Apache Beam runner I mentioned above, Beam is notably lacking a performant and user-friendly single-machine multi-processing runner. At a minimum, I believe this PR (or something like it), would unblock the Dask backend work there, e.g. by unblocking https://github.com/apache/beam/issues/29365 for single-machine multiprocessing.
Very curious to hear your thoughts. Thanks so much.

Thanks for weighing in, @TomAugspurger.
Of course will defer to dask folks re: deprecation or not.

Just a gentle ping to ask if merging this as an incremental improvement seems like a possibility, or if there would be further changes needed before that happens?
This would still be very impactful to our work on Pangeo Forge if/when it's able to go in. Thanks so much!

Thanks so much, @fjetter !


Fix "invalid escape sequences" error on Python 3.6

Before submitting a pull request make sure you have:

[x] At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
[x] Searched the bugtracker for similar pull requests

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[x] I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[x] Bug fix
[ ] Improvement
[ ] New extractor
[ ] New feature

Description of your pull request and other information
Since Python 3.6, invalid escape sequences are invalid. An error occurs only if -W error is used. For example:
$ python -W error -m youtube_dl
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.6/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.6/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/home/yen/Projects/youtube-dl/youtube_dl/__init__.py", line 15, in <module>
    from .options import (
  File "/home/yen/Projects/youtube-dl/youtube_dl/options.py", line 8, in <module>
    from .downloader.external import list_external_downloaders
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/__init__.py", line 3, in <module>
    from .common import FileDownloader
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/common.py", line 10, in <module>
    from ..utils import (
  File "/home/yen/Projects/youtube-dl/youtube_dl/utils.py", line 504
    path_part if path_part in ['.', '..'] else re.sub('(?:[/<>:"\\|\\\\?\\*]|[\s.]$)', '#', path_part)
                                                             ^
SyntaxError: invalid escape sequence \s

This pull request fixes them all. Only broken strings are fixed. Regular expressions without invalid escape sequences are left untouched.

References:

http://bugs.python.org/issue27364: Deprecate invalid escape sequences in str/bytes
http://bugs.python.org/issue25677: Syntax error caret confused by indentation

Thanks for the feedback. Bilibili change reverted.

LGTM.

Landed as ec85ded.


Covid chatbot pmidc team

Update RRT Flow For ChatBot @ramakrishnaippili-eGov @rakesh-wt-egov @rushang7-eGov

@egovmis1
Raise the PR to "covid-chatbot-pmidc-team" branch. The "chatbot-covid-punjab-2021" branch is reserved for production ready code.


chore(deps): update all the dependencies

#142
#141  👈 (View in Graphite)
next

This stack of pull requests is managed by Graphite. Learn more about stacking.


new: add a drop_pct referred to the global number of events

What type of PR is this?
/kind cleanup
/kind feature
Any specific area of the project related to this PR?
/area engine
What this PR does / why we need it:
Looking at this PR from @shane-lawrence I thought that we need a little bit of cleanup here, and moreover, it could be useful to have the drop_percentage computed on the global statistics and not only on the delta. I used the nlohmann json library just to simplify the output :) Let me know what you think about that
Which issue(s) this PR fixes:
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
new: add a `drop_pct` referred to the global number of events

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

👍👍👍

/milestone 0.33.0

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP, jasondellaluce
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,jasondellaluce]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


IKEA light bulbs acting weird

I am using Conbee II as my bridge. I have created a scene in Homekit. It sets the bulbs to different values (sometimes) and I have to click the scene multiple times before it's correctly set.
Not sure what's going on there.
Have a look: https://youtu.be/9MiVmj2e6C4

Please capture and attach a debug log file, see https://github.com/ebaauw/homebridge-hue#debug-log-file.

Will do

I have yet to provide you with a log, however, I will give it soon.
I do, however, consider going back to the Hue Bridge and then only using the deCONZ/Procons GW for stuff such as IKEA sensors / Phillips Hue Switches (to get them to Homekit) and then having the rest in Hue. Do you have any experience / recommendations regarding this approach?
It seems like the Hue Bridge might be more stable than deCONZ. I saw that you posted an issue on their GitHub some months ago and no real response has come from their developers so I fear it's not as well maintained.
What are your thoughts @ebaauw

It seems like the Hue Bridge might be more stable than deCONZ.

I don't think that wording is correct, but certainly, the typical Zigbee network formed by a Hue bridge is more stable than the typical Zigbee network formed by deCONZ.  The stability issues are due to network size and combining Zigbee devices from different vendors.  People move to deCONZ because they want to use devices that aren't supported by the Hue bridge and/or because they want to overcome the (practical) size limitations imposed by the Hue bridge.

I saw that you posted an issue on their GitHub some months ago and no real response has come from their developers so I fear it's not as well maintained.

Check the Hue developers forum for Hue bridge issues that haven't been addressed for five years or longer (I only joined in 2014).  Most prominently: the lack of push notifications by the Hue bridge.  The deCONZ REST API is open source, so at least you can do something yourself, addressing issues.  Of course, when there's a viable workaround, there's less priority to address an issue.

I do, however, consider going back to the Hue Bridge and then only using the deCONZ/Procons GW for stuff such as IKEA sensors / Phillips Hue Switches (to get them to Homekit) and then having the rest in Hue. Do you have any experience / recommendations regarding this approach?

I've been running deCONZ stably for years, over 100 devices, half routers, half end-devices.  I recently split off my Eurotronic Spirit TRVs and my IKEA FYRTUR smart blind and controller to a second Zigbee network, formed by a second installation of deCONZ, because they kept becoming unreachable.  I've had no issues with these devices since.  I use a single IKEA repeater on the second network for reach, otherwise it's just the end-devices.
Splitting your network between lights and controllers might not be the optimal design:

The end-devices need routers (lights) for connectivity, but you might use some repeaters instead (as I have).
You can no longer control the lights directly from the controller (so they work even when the gateway is down).
You can no longer use gateway/bridge rules to control the lights.  I'm experimenting with web hooks from deCONZ rules, to update a CLIP sensor on the other gateway (or Hue bridge), triggering rules on that gateway/bridge.  It seems to work reliably, but is somewhat limited as you need a separate rule for each sensor value.  I wouldn't recommend this approach to non-tech-savvy people.

I would rather split by vendors that seem to work well together.  I still haven't been able to find the source of my reachability issues; it could be some Xiaomi and innr routers still in my network; it could be the end-devices themselves.  Splitting off the end-devices was easier for me, as they're controlled by the Daylight sensor and local time, which are available on each gateway/bridge.

Thanks for your throughout answer. I will share some dump logs as soon as I can.

I can confirm issues with scenes and Ikea bulbs too. I try to support you with a log too. In my case switching on and off and setting the light spectrum works but not dimming. Dimming up or down is mostly ignored. For example the light is at 80% but HomeKit and the DeCONZ Web UI is showing it at 1%. Reading the node info in the VNC Session gives 80%. I looks like it gets lost somewhere ...

I created two Scenes "Test Tag" and "Test Nacht":
"Test Tag": "Deckenlampe Büro" gets Cool and Bright, "Tischlampe Büro" gets Cool and off
"Test Nacht": "Deckenlampe Büro" gets Warm and Dark, "Tischlampe Büro" gets Warm and Bright
Switching between the scenes changes Cool/Warm every time but the brightness and on/off-state are randomly not triggered (often after the third activation of the scene"
Here are screenshot from switching through the scenes:

Here you can see that the Deckenlampe Büro changed from 95% to 1% but stays at Level 241

Here you can see that the Deckenlampe Büro changed from 1% to 95% but stays at Level 3

@ebaauw here is the log:
homebridge.log.gz

I think the problems are the ignoring of settings:

No that's completely unrelated.
The issue is caused by a firmware bug in the IKEA lights, see https://github.com/dresden-elektronik/deconz-rest-plugin/issues/3507.  From the HomeKit scene, you set both brightness and colour temperature, which Homebridge Hue translates into a single API call (per light) setting both ct and bri.  deCONZ (or the Hue bridge) translates this into two separate Zigbee commands: Move (to the Level Control cluster) and Move to Color Temperature (to the Color Control cluster), adding a default transition time of 0.4 seconds to each command.  Due to the firmware bug, (some?) Trådfri lights don't accept the second command, while the transition from the first command is still in progress.
This needs to be solved by IKEA in the light firmware.  Until then, there are a couple of possible workarounds:

Make Homebridge Hue add a transitiontime of 0 to the API call, which deCONZ/Hue bridge passes to each Zigbee command, causing the light to transitions instantly.  You do this in Eve, by adding the Transition Time characteristic from the gateway/bridge accessory to the HomeKit scene.  Homebridge Hue will reset this to 0.4s after the scene has been recalled.  This workaround will effect all lights in the scene: you'll notice a blunt change, rather than the more subtle transition you're used to.
Make Homebridge Hue send two separate API calls, hoping that this will cause enough delay for the second command to arrive at the light 0.4s or more after the first command.  You do this by setting waitTimeUpdate to 0 (ms) in config.json.  This global setting will effect all HomeKit scenes and might cause delays in lights reacting when recalling a scene.
Use Zigbee scenes instead.  Set both groups and scenes in config.json to expose these to HomeKit, see #475.

It might be needed to combine 1 and 2, e.g. setting a transitiontime of 0.2s in combination with a 0ms waitTimeUpdate.  It might also be enough just to set the transitiontime to 0.1s or 0.2s.

Make Homebridge Hue add a transitiontime of 0 to the API call, which deCONZ/Hue bridge passes to each Zigbee command, causing the light to transitions instantly. You do this in Eve, by adding the Transition Time characteristic from the gateway/bridge accessory to the HomeKit scene.

I tried it and it works. The scene changes insane fast 😅

Make Homebridge Hue send two separate API calls, hoping that this will cause enough delay for the second command to arrive at the light 0.4s or more after the first command. You do this by setting waitTimeUpdate to 0 (ms) in config.json.

Where is the right spot to set waitTimeUpdate in the config.json? I want to try the combined settings.

Where is the right spot to set waitTimeUpdate in the config.json? I want to try the combined settings.

In the platform section for Homebridge Hue.  Best use the plugin's Settings dialogue in Homebridge Config UI X (under Advanced Settings, obviously).

waitTimeUpdate doesn't help. Transition Time 0 is the only suitable solution for me at this time. Other values or mixed values of solution 1 and 2 doesn't work correctly.

Can you verify in the debug log, that Homebridge Hue sends separate commands for ct and bri with waitTimeUpdate?

with waitTimeUpdate the commands ct bri and on are transmitted separately but the lights react still crazy:
[37m[20.11.2020, 07:56:01] [Hue] Phoscon-GW: gateway request 23: put /lights/15/state {"bri":3}
[20.11.2020, 07:56:01] [Hue] Tischlampe Büro: homekit brightness changed from 90% to 56%
[20.11.2020, 07:56:01] [Hue] Phoscon-GW: gateway request 24: put /lights/17/state {"bri":142}
[20.11.2020, 07:56:01] [Hue] Deckenlampe Büro vorne: homekit color temperature changed from 250 mired to 413 mired
[20.11.2020, 07:56:01] [Hue] Phoscon-GW: gateway request 25: put /lights/15/state {"ct":413}
[20.11.2020, 07:56:01] [Hue] Tischlampe Büro: homekit on changed from 0 to 1
[20.11.2020, 07:56:01] [Hue] Phoscon-GW: gateway request 26: put /lights/17/state {"on":true}
[20.11.2020, 07:56:01] [Hue] Tischlampe Büro: homekit color temperature changed from 390 mired to 454 mired
[20.11.2020, 07:56:01] [Hue] Phoscon-GW: gateway request 27: put /lights/17/state {"ct":454}
[20.11.2020, 07:56:01] [Hue] Phoscon-GW: gateway request 26: ok
[20.11.2020, 07:56:01] [Hue] Phoscon-GW: gateway request 23: ok
[20.11.2020, 07:56:01] [Hue] Phoscon-GW: gateway request 25: ok
[20.11.2020, 07:56:01] [Hue] Phoscon-GW: gateway request 24: ok
[20.11.2020, 07:56:01] [Hue] Phoscon-GW: gateway request 27: ok
[20.11.2020, 07:56:01] [Hue] Tischlampe Büro: state changed event: {"alert":null,"bri":142,"colormode":"ct","ct":454,"on":true,"reachable":true}
[20.11.2020, 07:56:01] [Hue] Deckenlampe Büro vorne: state changed event: {"alert":null,"bri":3,"colormode":"ct","ct":413,"on":true,"reachable":true}
[20.11.2020, 07:56:01] [Hue] Tischlampe Büro: state changed event: {"alert":null,"bri":142,"colormode":"ct","ct":454,"on":true,"reachable":true}
[20.11.2020, 07:56:01] [Hue] Tischlampe Büro: state changed event: {"alert":null,"bri":207,"colormode":"ct","ct":454,"on":true,"reachable":true}
[20.11.2020, 07:56:01] [Hue] Tischlampe Büro: light bri changed from 142 to 207
[20.11.2020, 07:56:01] [Hue] Tischlampe Büro: recently updated - ignore changed bri
homebridge.log.gz

Homebridge Hue is still sending the requests in parallel, i.e. it’s sending the next request before the previous request has returned.  You could try and set parallelRequests to 1 in combination with waitTimeUpdate.  Again, this will impact overall responsiveness; using Transition Time looks weird, but, indeed, works.

I tried parallelRequests 1 and waitTimeUpdate 0. There are still commands ignored by the lights 😞

@simplenotezy I made a workaround for the scenes in home. I execute 2 different scenes - one changes only brightness and on/off-state and a second changes the colourtemp. By using virtual automation switches you can trigger this by one click.
Example:
For my day ambience I have a virtual switch "Tagmodus Büro" and two scenes "Büro Tag Helligkeit" and "Büro Tag Farbe". I use a transition time in each scene of around 9 Seconds. I wrote a home-automation which is triggered by turning the virtual "Tagmodus Büro"-switch to on. The automation executed the two scenes with a delay of 10 seconds between each scene. After the execution I switch the virtual switch to off again.
Here is the home-automation:

Here is the scene definition in eve:

For the virtual switch I use https://github.com/grover/homebridge-automation-switches#readme.
Here is a example config for the switch:
      {             "platform": "AutomationSwitches",             "switches": [                 {                     "type": "switch",                     "name": "Tagmodus Büro",                     "stored": true,                     "default": false                 },                 {                     "type": "switch",                     "name": "Nachtmodus Büro",                     "stored": true,                     "default": false                 }             ]         }


Fix: Ed is stripped in LaTeX formats

Ref: https://github.com/cern-sis/issues-inspire/issues/53

@nooraangelva thank you for your PR, I believe it would be cleaner if we do the following:

Parametririze https://github.com/inspirehep/inspire-utils/blob/54cd0488d511fa2f1090efefcd437d9ea3a72ab4/inspire_utils/name.py#L44-L59 to include or not the titles

On https://github.com/inspirehep/inspire-utils/blob/54cd0488d511fa2f1090efefcd437d9ea3a72ab4/inspire_utils/name.py#L477-L495 add parametririze to initialize ParsedName with contstant without titles, we can pass it in the init function https://github.com/inspirehep/inspire-utils/blob/54cd0488d511fa2f1090efefcd437d9ea3a72ab4/inspire_utils/name.py#L80-L81

On hep just call format_names(name, without_titles=True) for that case and that's all 🎉


Live server

My live server is latest version and it is not working. When I am typing code it's not updating preview in browser. To see the changes I need to refresh browser manually. What's the solution of this problem? Please help me.

What Operating System are you on?


Electron crash on Arch Linux with "GPU process isn't usable. Goodbye" - this fix works for me:

In vendor/astilectron/main.js, I added this line just above start();
app.commandLine.appendSwitch('in-process-gpu');

what i can see this is this file, https://github.com/asticode/astilectron/blob/master/main.js, correct? what i know i'm unable to modify this from this repo, but i'll see if i can add that flag in another way, or if you have a suggestion

closing due to no response


[website] showcase page: Google adwords app link isn't valid anymore

From this Travis log:
- ./_site/showcase/index.html
  *  External link https://adwords.google.com/home/resources/adwords-app.html failed: 404 No error

cc @kwalrath @filiph @Sfshaza

It might be because of rebranding:

AdWords is now Google Ads. Our new name reflects the full range of advertising options we offer across Search, Display, YouTube, and more. Learn more

Maybe someone from Google Ads can tell us where to find the app?

Closed by https://github.com/flutter/website/pull/1119


view not updating when elements changed in loop

Description:
I upgraded to 0.8.3 from 0.7.3 and I've noticed some unexpected behavior.  If I set an array on ractive and then later add items to that array asynchronously (say after an ajax call), Ractive used to update the view layer as I pushed those objects.  Now I have to explicitly set the variable on ractive (or call ractive.update) to update the view.
I was unsure if this is expected behavior or if I needed to set a configuration to use the same behavior that was in 0.7.3.   I'm also not sure if this is a bug, but I've encountered this in other areas, and have either had to rewrite code, or force the view layer to update with ractive.update()
Versions affected:
Ractive 0.8.3
Ractive 0.8.1
Code:
      find: function() {
            var _this = this;
            var search = this.get('search');

            var _id = uuid();
            var out = [];

            $.ajax({
                url: "/ajaxurl/",
                data: {
                    q: search
                }
            }).done(function (data) {
                data.results.forEach(function (result) {
                    out.push(result);
                });

                // _this.set('searchResults', out); , <- this will update the view layer with 0.8.3 and 0.7.3
            });

            this.set('searchResults', out); <- this will only update the view layer with 0.7.3;  0.8.1 and 0.8.3 will not update
        }

This is intentional and listed as a breaking change in the changelog, I think. You can get the old behavior back by passing modifyArrays: true as an init option. The reason modifyArrays is now defaulted to false is that it's a bit heavier than you would expect, and it is set to be pulled out of the main code base into a plugin in a future release. magic will also be pulled into a plugin.
The idiomatic way to handle this would be to use this.push('searchResults', result) for each data result. Or even better, skip the forEach and use
// put the keypath to push to as the first arg
data.results.unshift('searchResults');
// apply the push with all results pushed at once, so only one update :)
this.push.apply(this, data.results);

Thanks for the super fast reply.  I read through here but it didn't register when I read through it.     This is pretty clear in the docs related to options, so I will close my issue.


Feat: Adding Top section to bottom section navigator

Describe the feature
I am thinking of adding a way through which when user visit the page then an moving down arrow will be shown and then when user clicked on that arrow the user will scrolled down to the next container or section .
Add ScreenShots

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[ ] I'm a VSoC'24 contributor
[X] I want to work on this issue

@Ayushmaanagarwal1211 go ahead


Edit mode prep

Safe changes to make now that should help with https://github.com/KittyCAD/engine/pull/2101

@Irev-Dev there's no more blockers on the engine side merge.  Lmk this week if you feel good about this and want to merge it in on both sides.

@mlfarrell Edit mode in the engine needs to be made a no-op, can we do that? This will allow us to update to the new behaviour without having to time merges.

@Irev-Dev edit mode is a no-op on my open edit mode removal PR (the one you likely tested this against)


bug: Unable to set enableBackdropDismiss to false for modals

Ionic version:  (check one with "x")
[ ] 1.x (For Ionic 1.x issues, please use https://github.com/ionic-team/ionic-v1)
[ ] 2.x
[X] 3.x
I'm submitting a ...  (check one with "x")
[X] bug report
[ ] feature request
[ ] support request => Please do not submit support requests here, use one of these channels: https://forum.ionicframework.com/ or http://ionicworldwide.herokuapp.com/
Current behavior:

let modal = modalCtrl.create(SomePage, {}, { enableBackdropDismiss: false });
modal.present();

In Chrome (Version 59.0.3071.115 (Official Build) (64-bit)), I inspected $ ionic serve and enabled the device toolbar through DevTools with iPad Mini selected.
The modal appears, but clicking on the backdrop, the modal closes.
Expected behavior:

The modal appears and clicking on the backdrop will not close the modal.
Steps to reproduce:

Using the latest Ionic (v3.5.3), I ran the application in the browser using $ ionic serve command and through Chrome DevTools, I inspected the application and turned on the Device Toolbar with the iPad Mini device selected.
I also added a log to ModalCmp class in the generated www/vendor.js file to ensure that the option is pulling through.
The this._bdDismiss private variable stays true and does not allow me to change the default option.
Related code:
function ModalCmp(_cfr, _renderer, _elementRef, _navParams, _viewCtrl, gestureCtrl, moduleLoader) {
        this._cfr = _cfr;
        this._renderer = _renderer;
        this._elementRef = _elementRef;
        this._navParams = _navParams;
        this._viewCtrl = _viewCtrl;
        this.moduleLoader = moduleLoader;
        var opts = _navParams.get('opts');
        (void 0) /* assert */;
        this._gestureBlocker = gestureCtrl.createBlocker({
            disable: [__WEBPACK_IMPORTED_MODULE_4__gestures_gesture_controller__["d" /* GESTURE_MENU_SWIPE */], __WEBPACK_IMPORTED_MODULE_4__gestures_gesture_controller__["b" /* GESTURE_GO_BACK_SWIPE */]]
        });
        this._bdDismiss = opts.enableBackdropDismiss;

        console.log('------------------------');
        console.log(_navParams.get('opts'));
        console.log(this._bdDismiss);

        if (opts.cssClass) {
            opts.cssClass.split(' ').forEach(function (cssClass) {
                // Make sure the class isn't whitespace, otherwise it throws exceptions
                if (cssClass.trim() !== '')
                    _renderer.setElementClass(_elementRef.nativeElement, cssClass, true);
            });
        }
    }

Other information:

Ionic info: (run ionic info from a terminal/cmd prompt and paste output below):
global packages:

    @ionic/cli-utils : 1.5.0
    Cordova CLI      : 7.0.1
    Ionic CLI        : 3.5.0

local packages:

    @ionic/app-scripts              : 2.0.2
    @ionic/cli-plugin-cordova       : 1.4.1
    @ionic/cli-plugin-ionic-angular : 1.3.2
    Cordova Platforms               : android 6.2.3
    Ionic Framework                 : ionic-angular 3.5.3

System:

    Node       : v6.10.3
    OS         : macOS Sierra
    Xcode      : Xcode 8.3.3 Build version 8E3004b
    ios-deploy : 1.9.1
    ios-sim    : 5.0.8
    npm        : 3.10.10

Hello, thanks for using Ionic! Unfortunately, I am not able to reproduce this issue. Would you mind posting a repo we could use to reproduce this issue?

@jgw96
Thanks.  I created a brand new project and could not re-create the issue.
Closing this issue.


RCE fix: replaced child_process execSync with spawnSync

📊 Metadata *
The npm-git-publish module is vulnerable against RCE due to user-supplied inputs being formatted and executed without proper validation
Bounty URL:  https://www.huntr.dev/bounties/1-npm-npm-git-publish/
⚙ Description *
Mitigating the arbitrary code execution by changing the execution means rather than using more library code for sanitizing.
💻 Technical Description *
Every call to "child_process.execSync" api is replaced by "child_process.spawn" apis. The spawn  method spawns a new process using the given command, with command-line arguments in args. If omitted, args defaults to an empty array. This mitigates the arbitrary code execution vulnerability.
🐛 Proof of Concept (PoC) *
Create the following PoC file:
// poc.js
var git = require('npm-git-publish');
git.publish('.', 'http://gihub.com ;touch HACKED; #')
Check there aren't files called HACKED
Execute the following commands in another terminal:
npm i npm-git-publish # Install affected module
node poc.js #  Run the PoC
Recheck the files: now HACKED has been created
🔥 Proof of Fix (PoF) *
Before:

After:

👍 User Acceptance Testing (UAT)

Hi @d3v53c 😄
thanks for the fix 👍
The line seems ok to me, but could you please fix also the issue in the following lines?:

https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L132
https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L181
https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L188
https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L193
https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L206

Also, I would suggest using execFile instead of spawn (please do let me know if you chose that one for a specific reason 👍 ) which avoids totally the RCE 😄
Cheers,
Mik

I chose spawn over execFile, because execFile runs the executable until it exits or terminates, then returns a buffer for data on stdout or stderr with a maximum size of 200Kb. spawn can stream stdout or stderr back to the parent process once it starts running, and there is no limit to the size of data it can return. Also, I'm a little biased towards spawn. 😄
These 2 functions apparently run in a largely similar manner, by executing a file in a child process. I didn't have any specific reason other than this. If you insist, I can change it to execFile, since in this case, it hardly matters, I guess.
Please let me know your thoughts too.
Cheers!!

Hi @d3v53c smile
thanks for the fix +1
The line seems ok to me, but could you please fix also the issue in the following lines?:

https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L132
https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L181
https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L188
https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L193
https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L206

Also, I would suggest using execFile instead of spawn (please do let me know if you chose that one for a specific reason +1 ) which avoids totally the RCE smile
Cheers,
Mik

Hi @Mik317, Good morning,
I chose spawn over execFile, because execFile runs the executable until it exits or terminates, then returns a buffer for data on stdout or stderr with a maximum size of 200Kb. spawn can stream stdout or stderr back to the parent process once it starts running, and there is no limit to the size of data it can return. Also, I'm a little biased towards spawn.
These 2 functions apparently run in a largely similar manner, by executing a file in a child process. I didn't have any specific reason other than this. If you insist, I can change it to execFile, since in this case, it hardly matters, I guess.
Please let me know your thoughts too.
Cheers!!

Thanks for the explanation :smile:
I think spawn is ok :+1:
As I previously said, if you can fix also the other lines which could present the vulnerability it would be awesome :100: (if you re-ask the review I can accept those modifications too)
Cheers,
Mik

Hi @d3v53c smile
thanks for the fix +1
The line seems ok to me, but could you please fix also the issue in the following lines?:

https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L132
https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L181
https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L188
https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L193
https://github.com/418sec/npm-git-publish/blob/8332bbbe32acce0331f1bf9a54b47acfcba51803/lib/publish.ts#L206

Also, I would suggest using execFile instead of spawn (please do let me know if you chose that one for a specific reason +1 ) which avoids totally the RCE smile
Cheers,
Mik

Hi @Mik317, Good morning,
I chose spawn over execFile, because execFile runs the executable until it exits or terminates, then returns a buffer for data on stdout or stderr with a maximum size of 200Kb. spawn can stream stdout or stderr back to the parent process once it starts running, and there is no limit to the size of data it can return. Also, I'm a little biased towards spawn.
These 2 functions apparently run in a largely similar manner, by executing a file in a child process. I didn't have any specific reason other than this. If you insist, I can change it to execFile, since in this case, it hardly matters, I guess.
Please let me know your thoughts too.
Cheers!!

Thanks for the explanation 😄
I think spawn is ok 👍
As I previously said, if you can fix also the other lines which could present the vulnerability it would be awesome 💯 (if you re-ask the review I can accept those modifications too)
Cheers,
Mik

Thank you @Mik317,
I'll update the PR once I have done with the requested changes, and I'll re-ask for your review too.
You're awesome!!
Advanced happy new year wishes to you and all..!!
Cheers!!

Thanks :heart:
Happy new year you too :+1:
Cheers,
Mik

@d3v53c - thanks for your PR.
This bounty was made live during our learning experiment on the platform where we encouraged our users to submit fixes for previously fixed vulnerabilities.
This vulnerability was previously fixed here...
Apologies for the delay in the response! 🍰

@d3v53c - thanks for your PR.
This bounty was made live during our learning experiment on the platform where we encouraged our users to submit fixes for previously fixed vulnerabilities.
This vulnerability was previously fixed here...
Apologies for the delay in the response! 🍰

Sorry d3v53c, we enjoyed reviewing your fix but it has not been selected this time. If this bounty has not been closed, please feel free to try again with a new pull request!
We appreciate your effort and look forward to reviewing more of your fixes in the future! 🔨😎
Come join us on Discord

Sorry d3v53c, we enjoyed reviewing your fix but it has not been selected this time. If this bounty has not been closed, please feel free to try again with a new pull request!
We appreciate your effort and look forward to reviewing more of your fixes in the future! 🔨😎
Come join us on Discord


UX lacking for tools management with go mod

This is the recommended way to track tools dependencies in Go involves parsing a tools.go:
https://marcofranssen.nl/manage-go-tools-via-go-modules/
That's a far cry from the tool management experience available today in Node.js, Python, and Ruby, where tools are already supported on par with libraries.
We could have a short term solution available very quickly, with some minor changes to the Go ecosystem to promote tools.go to an official standard:

golint, go vet, etc. would accept applications in imports.
go mod tidy would install any applications noted in tools.go.
go mod tidy and go install would improve support and documentation for specifying any custom build tags.

I'm not sure if tools.go is a long term strategy for tracking tools dependencies, but it's a start. Can we please get a better developer workflow than parsing Go code with awk?

There has been lots of discussion in this space in the last two years, such as https://github.com/golang/go/issues/30515, https://github.com/golang/go/issues/34506, and https://github.com/myitcv/gobin.
Is what you're suggesting a proposal? It seems a bit all over the place; for example, I hardly imagine go mod tidy will ever build or install anything. Also, golint doesn't play a part in the go tool.
If this is a proposal, it needs to take into account the previous work. If it's not a proposal, please clarify what exactly the issue is about.


Change network

This PR aims to add the change network feature mentioned in #46.
What's been done:

Modal for react-ui
Custom Select component for react-ui
Context and provider for react-ui modal
Update ConnectionProvider to track endpoints and update connection

What's being worked on:

Standalone button for switching networks without a wallet connection

Check react-ui starter project on how to use the new NetworkModalProvider

@jordansexton could you take a look at this PR?

@jordansexton and @tyirenkyi how close is this to being merged? We have a network selector in our app, and i'm trying to start using the wallet-adapter for it too, and this is the last weird sync area :D

Not at all close. I wouldn't count on it landing, and it's not a high priority. If you need network selection, better to implement in your app.

mmmm, darn :) thanks.

Closing as we probably won't implement this at the application or adapter level.
We will soon move away from requiring adapters for every wallet, and the new wallet standard will have a network param in the relevant methods.

Folks, could you point me to the new wallet standard? We have several applications deploying with custom RPCs, and while we could create a new standard so they can support this feature to change networks, we'd prefer to adopt what y'all already have in mind. It is cited as one of the most popular pain points for us.
It sounds like Backpack and Solflare are on board to implement whatever is needed on their end if we can surface the relevant information via the wallet adapter or however else.

@neelsomani https://github.com/solana-labs/wallet-standard/tree/master/packages/core/features is where this would likely go. Please open an issue on that repo before sending a PR though.
For context, Backpack currently implements this as window.backpack.connection which uses a web3.js Connection object. This is probably a higher level interface than we'd want to expose.
We've avoided dependency on web3.js across the Wallet Standard, using raw bytes everywhere instead of web3.js types. Also consider that web3.js is getting a planned rewrite.


current package contains test logs

test/.kitchen/ and test/.tmp/ probably don't belong in the released package of the module. Unfortunately, they're currently in the version at https://forge.puppetlabs.com/computology/packagecloud

oh, and modules/ and that .rpm file

+1 Would be great if the fixed version could be uploaded to the forge

@ice799 unfortunately, I can't solve this with a PR, this is entirely an issue with the process being used to build & upload to the forge.

I can't seem to find a way to exclude certain files/paths/regexps when creating a puppet module, so I'm still trying to figure out how to do this. Not sure where the .rpm file comes from, but one thing at a time.

I just uploaded 0.2.11 that I think fixed this. Please reopen this issue if this is not fixed, but a quick look seems to suggest its fixed now.


Discuss test_download_and_delete_fsaverage_surfaces()

Hi @Cai - Can we discuss the test test_download_and_delete_fsaverage_surfaces() at some point next week?
It is sensibly testing the downloading of the fsaverage surfaces. I suppose it doesn't matter too much but ... I think this requires it to download 250MB+196MB (i.e almost 0.5GB) by github actions every time we push a commit. I just wanted to make sure that we agreed whether this is sensible.

Interesting - this test makes sure that the reliance on downloading from MNE keeps working, but it seems mad to do this each time we push a commit. Maybe we can somehow split this off into a separate CI action which runs like once a month instead? Or just rely on it breaking for us in use rather than test it with CI.

Can we exclude certain tests from the on-push CI?

yes, we can with:
IN_GITHUB_ACTIONS = os.getenv("GITHUB_ACTIONS") == "true"

@pytest.mark.skipif(IN_GITHUB_ACTIONS, reason="Test doesn't work in Github Actions.")

Fixed.


CentOS 8 and Fedora 31/32 Cloud Images Ignore Network Configuration Files

Required information

Distribution: Ubuntu
Distribution version: 20.04
The output of "lxc info" or if that fails:

Kernel version:
LXC version:
LXD version:
Storage backend in use:

config:
cluster.https_address: <IP_ADDRESS>:8443
core.https_address: <IP_ADDRESS>:8443
core.trust_password: true
api_extensions:

storage_zfs_remove_snapshots
container_host_shutdown_timeout
container_stop_priority
container_syscall_filtering
auth_pki
container_last_used_at
etag
patch
usb_devices
https_allowed_credentials
image_compression_algorithm
directory_manipulation
container_cpu_time
storage_zfs_use_refquota
storage_lvm_mount_options
network
profile_usedby
container_push
container_exec_recording
certificate_update
container_exec_signal_handling
gpu_devices
container_image_properties
migration_progress
id_map
network_firewall_filtering
network_routes
storage
file_delete
file_append
network_dhcp_expiry
storage_lvm_vg_rename
storage_lvm_thinpool_rename
network_vlan
image_create_aliases
container_stateless_copy
container_only_migration
storage_zfs_clone_copy
unix_device_rename
storage_lvm_use_thinpool
storage_rsync_bwlimit
network_vxlan_interface
storage_btrfs_mount_options
entity_description
image_force_refresh
storage_lvm_lv_resizing
id_map_base
file_symlinks
container_push_target
network_vlan_physical
storage_images_delete
container_edit_metadata
container_snapshot_stateful_migration
storage_driver_ceph
storage_ceph_user_name
resource_limits
storage_volatile_initial_source
storage_ceph_force_osd_reuse
storage_block_filesystem_btrfs
resources
kernel_limits
storage_api_volume_rename
macaroon_authentication
network_sriov
console
restrict_devlxd
migration_pre_copy
infiniband
maas_network
devlxd_events
proxy
network_dhcp_gateway
file_get_symlink
network_leases
unix_device_hotplug
storage_api_local_volume_handling
operation_description
clustering
event_lifecycle
storage_api_remote_volume_handling
nvidia_runtime
container_mount_propagation
container_backup
devlxd_images
container_local_cross_pool_handling
proxy_unix
proxy_udp
clustering_join
proxy_tcp_udp_multi_port_handling
network_state
proxy_unix_dac_properties
container_protection_delete
unix_priv_drop
pprof_http
proxy_haproxy_protocol
network_hwaddr
proxy_nat
network_nat_order
container_full
candid_authentication
backup_compression
candid_config
nvidia_runtime_config
storage_api_volume_snapshots
storage_unmapped
projects
candid_config_key
network_vxlan_ttl
container_incremental_copy
usb_optional_vendorid
snapshot_scheduling
container_copy_project
clustering_server_address
clustering_image_replication
container_protection_shift
snapshot_expiry
container_backup_override_pool
snapshot_expiry_creation
network_leases_location
resources_cpu_socket
resources_gpu
resources_numa
kernel_features
id_map_current
event_location
storage_api_remote_volume_snapshots
network_nat_address
container_nic_routes
rbac
cluster_internal_copy
seccomp_notify
lxc_features
container_nic_ipvlan
network_vlan_sriov
storage_cephfs
container_nic_ipfilter
resources_v2
container_exec_user_group_cwd
container_syscall_intercept
container_disk_shift
storage_shifted
resources_infiniband
daemon_storage
instances
image_types
resources_disk_sata
clustering_roles
images_expiry
resources_network_firmware
backup_compression_algorithm
ceph_data_pool_name
container_syscall_intercept_mount
compression_squashfs
container_raw_mount
container_nic_routed
container_syscall_intercept_mount_fuse
container_disk_ceph
virtual-machines
image_profiles
clustering_architecture
resources_disk_id
storage_lvm_stripes
vm_boot_priority
unix_hotplug_devices
api_filtering
instance_nic_network
clustering_sizing
firewall_driver
projects_limits
container_syscall_intercept_hugetlbfs
limits_hugepages
container_nic_routed_gateway
projects_restrictions
custom_volume_snapshot_expiry
volume_snapshot_scheduling
trust_ca_certificates
snapshot_disk_usage
clustering_edit_roles
container_nic_routed_host_address
container_nic_ipvlan_gateway
resources_usb_pci
resources_cpu_threads_numa
resources_cpu_core_die
api_os
container_nic_routed_host_table
container_nic_ipvlan_host_table
container_nic_ipvlan_mode
resources_system
images_push_relay
network_dns_search
container_nic_routed_limits
instance_nic_bridged_vlan
network_state_bond_bridge
usedby_consistency
custom_block_volumes
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
tls
environment:
addresses:

<IP_ADDRESS>:8443
architectures:
x86_64
i686
certificate: |
-----BEGIN CERTIFICATE-----
MIICODCCAb+gAwIBAgIQYcYTiR+fnnzFdXzMyC3EYzAKBggqhkjOPQQDAzBGMRww
GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMSYwJAYDVQQDDB1yb290QGNsb3Vk
Ny5jbG91ZGJyb2NrdGVjLmNvbTAeFw0yMDA1MjQxNjU1MjFaFw0zMDA1MjIxNjU1
MjFaMEYxHDAaBgNVBAoTE2xpbnV4Y29udGFpbmVycy5vcmcxJjAkBgNVBAMMHXJv
b3RAY2xvdWQ3LmNsb3VkYnJvY2t0ZWMuY29tMHYwEAYHKoZIzj0CAQYFK4EEACID
YgAEyYXxdQTDltvWDJHdfDZgNBDEtV58mWH0NIK6VdDwxMpeagS2WenPz7F4wBHH
RVw33wltFY74jgOFhDpYDvGd/MXNkmNpwIXP+pifqzB8N26MNEAN1D3LW+sYuF7T
4/nYo3IwcDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYD
VR0TAQH/BAIwADA7BgNVHREENDAyghhjbG91ZDcuY2xvdWRicm9ja3RlYy5jb22H
BH8AAAGHEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDZwAwZAIwEfEycl0i
tmpHUSzK5nFCBAW1+rMP3zaNKsN8nMwnmtmrWGX1Yl1r4yQcnrTx3hQgAjAyp9bU
qVYY00N5Mb2yxX1Nj0v8X+R1UbSa+ZZmyoN4CGK3hHysWBr2tu3fZOfEv2o=
-----END CERTIFICATE-----
certificate_fingerprint: f5977db2c09dc64907f4b1911bc8f4bfca571ecd77ca1db2e5936b0fcd42ac7f
driver: lxc
driver_version: 4.0.3
firewall: nftables
kernel: Linux
kernel_architecture: x86_64
kernel_features:
netnsid_getifaddrs: "true"
seccomp_listener: "true"
seccomp_listener_continue: "true"
shiftfs: "false"
uevent_injection: "true"
unpriv_fscaps: "true"
kernel_version: 5.4.0-40-generic
lxc_features:
cgroup2: "true"
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
network_veth_router: "true"
pidfd: "true"
seccomp_notify: "true"
os_name: Ubuntu
os_version: "20.04"
project: default
server: lxd
server_clustered: true
server_name: cloud10
server_pid: 2293
server_version: "4.3"
storage: zfs
storage_version: 0.8.3-1ubuntu12

Issue description
A brief description of the problem. Should include what you were
attempting to do, what you did, what happened and what you expected to
see happen.
Steps to reproduce

Start with a bare bones Ubuntu 20.04 system and the latest LXD Snap
Create an LXC Profile called "vlan13" with a macvlan nictype and a cloud init static ip
Create a CentOS 8 or Fedora 32 Container Instance using that profile
lxc launch images:centos/8/cloud -p default -p vlan13 ipa1
Note that /etc/sysconfig/network-scripts/ifcfg-eth0 is populated with your static IP
Note that "ip addr" doesn't show eth0 as being up or your ip as being applied.

Information to attach

[ ] Any relevant kernel output (dmesg)
[ ] Container log (lxc info NAME --show-log)
[ ] Container configuration (lxc config show NAME --expanded)
[ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
[ ] Output of the client with --debug
[ ] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

That would be a bug in NetworkManager or other distro scripts in CentOS/Fedora, not something that we have any control over in LXD.

So where do I file the bug request at?

LXD takes the config you provide and feeds it to cloud-init, the job of generating the config is cloud-init's job, the job of applying that config to your interfaces is NetworkManager's.
Note that if your interface isn't named eth0, you may need to duplicate one workaround we've put in place in the images.
If that's the case for you, look at and adapt /etc/systemd/system/network-device-down.service which is a workaround for a NetworkManager bug where it doesn't properly handle interfaces that are already up.

Most likely, against NetworkManager, either upstream or in the distro's bugtracker.


transfer learning ljspeech

First of all thank you for the repository, the result is really awesome.
Is it possible to transfer learning from pretrained models (ljspeech) into my recording data?

If it is english it is possible, but requires some code. You have to modify train-text2mel.py and train-ssrn.py to include such code:
text2mel.load_state_dict(torch.load('pretrained_ljspeech_text2mel.pth').state_dict())

and
ssrn.load_state_dict(torch.load('pretrained_ljspeech_ssrn.pth').state_dict())

Where would this piece of code be added?


Unique Key for Repeatable Livewire Component

I have a use case where i need to repeat / re-use the same Livewire Component in a single page.
I have noticed in the return of the livewire embed call it is targeting only the data-component.
I proposed to have a data-key to come along with the current data-component and data-params in order to replace the correct component in the HTML.

@adrianlzx1996 you might be able to create your "parent" embeddable component view with the correct number of nested components (in a loop for example) to achieve the desired effect ?

@adrianlzx1996 you might be able to create your "parent" embeddable component view with the correct number of nested components (in a loop for example) to achieve the desired effect ?

Correct me if i'm wrong, you're suggesting to put the repeatable component inside of a "parent" embeddable component, so that i do not have to repeat the child component in my web page?
This is entirely possible, however, it will create more hassle on my end to create the "parent" component, where the parent component have a complicated logic written and proven working, and i do not wish to re-write the parent component...

@adrianlzx1996 you might be able to create your "parent" embeddable component view with the correct number of nested components (in a loop for example) to achieve the desired effect ?

Correct me if i'm wrong, you're suggesting to put the repeatable component inside of a "parent" embeddable component, so that i do not have to repeat the child component in my web page?
This is entirely possible, however, it will create more hassle on my end to create the "parent" component, where the parent component have a complicated logic written and proven working, i do not wish to re-write the parent component...

Yes, the parent component becomes a container for your multiple (nested) components.  Livewire makes this very easy: https://livewire.laravel.com/docs/nesting#rendering-children-in-a-loop and also includes details on how to ensure your nested components can be reactive to any changes in the parent properties.

And dont forget this from the wire-extender documentation : If you are using nested components, please make sure all components have the #[Embeddable] trait.

I also have a use case for wanting to embed multiple components on a page, however it's not necessarily practical to have them 'nested'. Eg. Having multiple contact forms, at different points throughout a sales landing page. 95% of the content is on the Wordpress end, with the laravel application capturing the leads. 🤓


Upgrade to BigInts

Update primary keys in PostreSQL to BigSerial and add support for working with BigInts in the backend side.

Won't fix it for now.
I don't think is needed right now if it's running locally. INTEGER can go up to<PHONE_NUMBER>.


Create new route.

Description
As a user I must be able to create a new learning route.
Acceptance criteria

The user must be able to create a course whenever they want.
The course must contain materials related to whatever the user needs to learn.
The course must be based on the answers to the form.

For this story, we try to create many courses as the user wants, to verify the creations of the routes
Also we tried to put material on the course and the user should be able to look at the link, and click, the redirect should be working correctly


etextContextFacetGraph not returning matching chunks

see http://purl.bdrc.io/lib/etextContentFacetGraph?LG_NAME=bo-x-ewts&L_NAME="spyod 'jug"
bdr:UT3JT13379_001_0001 has matching chunks like bdr:EC34F7DBE013DECFF2 but they are not returned by query
(in this query all matching chunks for bdr:UT23703_1415_0000 are returned though)

moved to https://github.com/buda-base/xmltoldmigration/issues/148

it's fixed now


fix: enable the server setup hook by default in a custom server

Does this exclude writer edit? I think it should be deactivated for writer edit but activated by default for run even with custom server.

Does this exclude writer edit? I think it should be deactivated for writer edit but activated by default for run even with custom server.

This excludes the writer edit command. This is handled in calling code.
On the other hand, if you mount the application in edition in a multi app application, the edition will have authentication activated. You will have to deactivate it explicitly.


[ERROR] Error: unable to verify the first certificate

When sigining in, i set proxy. But i get an error [ERROR] Error: unable to verify the first certificate

Guess dup with #370

So, do you have idea to solve this?

So far I have no idea about this kind of issue.

Actually, i am also behind the proxy. But i have set proxy, and get the same error

Do you have the same error if you run the LeetCode CLI directly?

Sorry, i do not try that. Is it necessary？

The extension use the LeetCode CLI as the backend. If this issue could not repro when running CLI directly. Then I can check what's the difference.
I cannot repro the issue from my side so I can not do it by myself.


Product measurement/characterization revision to the schema

Resolves #432 (where most discussion has been), resolves #386, resolves #420, resolves #426
For discussion here:

Is it redundant to have both "counts" and "intensity"? I feel like the two aren't completely interchangeable, but there is certainly ambiguity.
Are we being completely consistent in terms of holding product-specific analytical data in this new ProductMeasurement field? In particular, I'm thinking about IR and HRMS of an isolated compound. The entire analysis is of the presumably-isolated product. Should ProductMeasurement accommodate a structured definition of the full IR spectrum? We shied away from this idea (#48) since it got rather complicated. However, HRMS is a very easy measurement to capture in a structured format -- the dominant peak is just a float m/z. It seems silly not to include this in the ProductMeasurement field. The temperature of a MP experiment or the angle of a CD experiment are similar.
To broaden the previous point, some analyses are applied to crude product mixtures, while others are applied to thought-to-be-pure compounds. in the former case, we definitely need this ProductMeasurement message to create a closer association between an analysis and a single product. But in the latter case, there is already an implicit connection. Some analyses result in a scalar value while others result in more structured data. One option is to add a Data field to the ProductMeasurement and a SPECTRUM type, but then there is ambiguity around when to use that field rather than the ReactionAnalysis's raw_data and processed_data maps. If we do add a Data field to ProductMeasurement, it would certainly be very handy for non-HTE data where people do full product characterizations; we would have access to many examples of structures and their NMR/IR with which to build predictive models.
I've kept color and texture in the ReactionProduct itself. They are not supposed to be tied to a specific analysis (unless we add "VISUAL_INSPECTION" as an analysis which is silly).
I don't think that uses_internal_standard and uses_authentic_standard need to be defined for the analysis, as they need to be defined for each (product, analysis) to handle the situation where multiple products are quantified and not all necessarily use an authentic or internal standard.

Note that this schema change breaks many things (hence the draft PR) and is not backwards compatible

Resolves #432 (where most discussion has been), resolves #386, resolves #420, resolves #426
For discussion here:

Is it redundant to have both "counts" and "intensity"? I feel like the two aren't completely interchangeable, but there is certainly ambiguity.

I don't have strong feelings about this, but I think if they are not interchangeable then keeping them separate is fine.

Are we being completely consistent in terms of holding product-specific analytical data in this new ProductMeasurement field? In particular, I'm thinking about IR and HRMS of an isolated compound. The entire analysis is of the presumably-isolated product. Should ProductMeasurement accommodate a structured definition of the full IR spectrum? We shied away from this idea (#48) since it got rather complicated. However, HRMS is a very easy measurement to capture in a structured format -- the dominant peak is just a float m/z. It seems silly not to include this in the ProductMeasurement field. The temperature of a MP experiment or the angle of a CD experiment are similar.

I'm not opposed to adding structured analysis messages piecemeal, especially if we can start with easy ones. We don't have to try to cover all the possibilities at once (and adding new ones to ProductMeasurement later will be backwards compatible).

To broaden the previous point, some analyses are applied to crude product mixtures, while others are applied to thought-to-be-pure compounds. in the former case, we definitely need this ProductMeasurement message to create a closer association between an analysis and a single product. But in the latter case, there is already an implicit connection. Some analyses result in a scalar value while others result in more structured data. One option is to add a Data field to the ProductMeasurement and a SPECTRUM type, but then there is ambiguity around when to use that field rather than the ReactionAnalysis's raw_data and processed_data maps. If we do add a Data field to ProductMeasurement, it would certainly be very handy for non-HTE data where people do full product characterizations; we would have access to many examples of structures and their NMR/IR with which to build predictive models.

This is indeed tricky. Maybe we need to think more broadly about how the analysis/product pairs should be structured. My current mental model is that an analysis may contain data that covers one or more compounds, and ProductMeasurement is the place to store compound-specific data extracted from an analysis. So I think that structured analysis data might be best placed in the ReactionAnalysis and summary statistics (like a specific peak area or quantified yield) are best placed in ProductMeasurement.
That doesn't really resolve what to do about single-compound analyses, like melting point, where the data could be in either place?

I've kept color and texture in the ReactionProduct itself. They are not supposed to be tied to a specific analysis (unless we add "VISUAL_INSPECTION" as an analysis which is silly).

SGTM.

I don't think that uses_internal_standard and uses_authentic_standard need to be defined for the analysis, as they need to be defined for each (product, analysis) to handle the situation where multiple products are quantified and not all necessarily use an authentic or internal standard.

Agreed; the placement in ProductMeasurement seems appropriate.

Note that this schema change breaks many things (hence the draft PR) and is not backwards compatible

Huzzah for alpha testing!

Hope to take a look this afternoon

Are we being completely consistent in terms of holding product-specific analytical data in this new ProductMeasurement field? In particular, I'm thinking about IR and HRMS of an isolated compound. The entire analysis is of the presumably-isolated product. Should ProductMeasurement accommodate a structured definition of the full IR spectrum? We shied away from this idea (#48) since it got rather complicated. However, HRMS is a very easy measurement to capture in a structured format -- the dominant peak is just a float m/z. It seems silly not to include this in the ProductMeasurement field. The temperature of a MP experiment or the angle of a CD experiment are similar.

I'm not opposed to adding structured analysis messages piecemeal, especially if we can start with easy ones. We don't have to try to cover all the possibilities at once (and adding new ones to ProductMeasurement later will be backwards compatible).

For IR, a similar field to HRMS but just repeated could be sufficient. If needed, an IR spectrum in a message could be captured by a repeated float value

To broaden the previous point, some analyses are applied to crude product mixtures, while others are applied to thought-to-be-pure compounds. in the former case, we definitely need this ProductMeasurement message to create a closer association between an analysis and a single product. But in the latter case, there is already an implicit connection. Some analyses result in a scalar value while others result in more structured data. One option is to add a Data field to the ProductMeasurement and a SPECTRUM type, but then there is ambiguity around when to use that field rather than the ReactionAnalysis's raw_data and processed_data maps. If we do add a Data field to ProductMeasurement, it would certainly be very handy for non-HTE data where people do full product characterizations; we would have access to many examples of structures and their NMR/IR with which to build predictive models.

A repeated float value of peak chemical shifts could almost capture an NMR spectrum too, except for 1H where string splitting information is also needed. We could try to build a single Spectrum message that could handle these, IR, and others with repeated float value and details fields, or could build specific messages for each expected spectrum type that isn't just floats

My current mental model is that an analysis may contain data that covers one or more compounds, and ProductMeasurement is the place to store compound-specific data extracted from an analysis.

I agree with this model, where ReactionAnalysis is used largely for mixtures or in-progress/crude analysis, and ProductMeasurement is used for single-component or purified/final compound analysis.

So I think that structured analysis data might be best placed in the ReactionAnalysis and summary statistics (like a specific peak area or quantified yield) are best placed in ProductMeasurement.

I think this is fine, but if "structured analysis data" is carried out on a purified compound or is being used to determine yield/identity of a product, I think I would prefer it be placed in ProductMeasurement (I guess like you're saying for "specific peak area or quantified yield").

That doesn't really resolve what to do about single-compound analyses, like melting point, where the data could be in either place?

As above, I personally think these single-compound analyses should be contained in ReactionProduct.

I've kept color and texture in the ReactionProduct itself. They are not supposed to be tied to a specific analysis (unless we add "VISUAL_INSPECTION" as an analysis which is silly).

Agreed.

I don't think that uses_internal_standard and uses_authentic_standard need to be defined for the analysis, as they need to be defined for each (product, analysis) to handle the situation where multiple products are quantified and not all necessarily use an authentic or internal standard.

Agreed; the placement in ProductMeasurement seems appropriate.

Right, sounds good.

Discussion notes:

Try to keep all original analytical data in one place -- the analysis message?
Add a boolean field to an analysis to say whether it's for a crude or (theoretically) purified product?
Storing structured analytical data for purified compounds isn't an explicit goal of this effort - let's focus on the primary reason for this modification, which is to allow us to distinguish relative/real yields and how other summary statistics (e.g., purity, selectivity) were derived with or without an internal standard
A boolean field in the analysis message would be helpful -- "is product-specific" or "is_of_isolated_species" to indicate that the analysis is supposed to correspond to a single species

New question for discussion, relevant to the raw_data and processed_data maps of an analysis: should we add arbitrary unit-ed fields here? For example, in addition to having float_value, string_value, etc., should we have Temperature temperature_value, Mass mass_value, etc.?

New question for discussion, relevant to the raw_data and processed_data maps of an analysis: should we add arbitrary unit-ed fields here? For example, in addition to having float_value, string_value, etc., should we have Temperature temperature_value, Mass mass_value, etc.?

Just to clarify, are you thinking about expanding the Data message or adding these to ReactionAnalysis?

New question for discussion, relevant to the raw_data and processed_data maps of an analysis: should we add arbitrary unit-ed fields here? For example, in addition to having float_value, string_value, etc., should we have Temperature temperature_value, Mass mass_value, etc.?

Just to clarify, are you thinking about expanding the Data message or adding these to ReactionAnalysis?

This would be an expansion of the Data message

New question for discussion, relevant to the raw_data and processed_data maps of an analysis: should we add arbitrary unit-ed fields here? For example, in addition to having float_value, string_value, etc., should we have Temperature temperature_value, Mass mass_value, etc.?

I like the idea of including these structured unit-ed / Data types here. Could open opportunity to add spectra / listed data forms later on

New question for discussion, relevant to the raw_data and processed_data maps of an analysis: should we add arbitrary unit-ed fields here? For example, in addition to having float_value, string_value, etc., should we have Temperature temperature_value, Mass mass_value, etc.?

I like the idea of including these structured / unit-ed Data types here. Could open opportunity to add spectra / listed data forms later on

I think I'm opposed to extending the Data message. It's designed to be the place to put things that have no other place---an unstructured "catch all" message. I recognize that there could be structured data that has no home, but I think that should be fixed with changes to the schema and not feature creep of Data.
I'd be more interested in adding these fields to ReactionAnalysis.

I'd be more interested in adding these fields to ReactionAnalysis.

SGTM

LGTM + readability

Thinking about migration: I think it will make migration easier if we don't delete anything from the schema. This way all of the old protos will continue to load properly and we can move things over without needing two copies of the repo (IIUC the current *.pbtxt messages won't parse after their fields are removed). It will be really ugly, but will make the migration much easier and then we'll clean everything up.
I'm thinking of something like this:
...
// BEGIN DEPRECATED
message Feature {	  map<string, Data> features = 15;
  string name = 1;	  // Compounds may be assayed for quality control; analytical data should be
  oneof kind {	  // defined in the analyses map.
    string string_value = 2;	  map<string, Analysis> analyses = 16;
    float float_value = 3;
  }
  // Details of computation. Software, theory level, solvation, etc.
  string how_computed = 4;
}
repeated Feature features = 13;
// END DEPRECATED
map<string, Data> features = 15;
// Compounds may be assayed for quality control; analytical data should be
// defined in the analyses map.
map<string, Analysis> analyses = 16;
...

Is that too much trouble?

Thinking about migration: I think it will make migration easier if we don't delete anything from the schema. ... Is that too much trouble?

No, I think that's reasonable. It will be very ugly, to be sure, but will make migration significantly easier. There will be some edge cases where it's not possible to do that because of field name conflicts (e.g., we're changing the type of Compound.features but keeping the name), but the majority of cases will be fine

Thinking about migration: I think it will make migration easier if we don't delete anything from the schema. ... Is that too much trouble?

No, I think that's reasonable. It will be very ugly, to be sure, but will make migration significantly easier. There will be some edge cases where it's not possible to do that because of field name conflicts (e.g., we're changing the type of Compound.features but keeping the name), but the majority of cases will be fine

I might rescind that answer. We can't do it for Compound.features or ReactionOutcome.analyses unless we want to choose different names for the fields, which I wouldn't advise.
I've added in the old schema as reaction_old.proto. This might seem sloppy, but we can at least make the transition since both versions will be importable in the same environment

Thinking about migration: I think it will make migration easier if we don't delete anything from the schema. ... Is that too much trouble?

No, I think that's reasonable. It will be very ugly, to be sure, but will make migration significantly easier. There will be some edge cases where it's not possible to do that because of field name conflicts (e.g., we're changing the type of Compound.features but keeping the name), but the majority of cases will be fine

I might rescind that answer. We can't do it for Compound.features or ReactionOutcome.analyses unless we want to choose different names for the fields, which I wouldn't advise.
I've added in the old schema as reaction_old.proto. This might seem sloppy, but we can at least make the transition since both versions will be importable in the same environment

SGTM. Thanks.

Here is a snippet from the updated Liu et al. OrgSyn example:
## Product and characterization
outcome = reaction.outcomes.add()
outcome.reaction_time.CopyFrom(unit_resolver.resolve('6 h'))
analysis = reaction_pb2.Analysis() # for handy enum reference

outcome.analyses['isolated_weight'].type = analysis.WEIGHT
outcome.analyses['isolated_weight'].is_of_isolated_species = True
outcome.analyses['isolated_weight'].details = '27.5-28.0 g recovered after workup'

outcome.analyses['1H NMR'].type = analysis.NMR_1H
outcome.analyses['1H NMR'].is_of_isolated_species = True
outcome.analyses['1H NMR'].details = '400 MHz, CDCl3'
outcome.analyses['1H NMR'].data['peaks'].string_value = r'0.92 (s, 9H), 4.06 (s, 4H), 7.23 - 7.34 (m, 6H), 7.40 (d, J = 7.1 Hz, 4H)'
outcome.analyses['1H NMR'].data['peaks'].description = 'List of peaks'

outcome.analyses['13C NMR'].type = analysis.NMR_13C
outcome.analyses['13C NMR'].is_of_isolated_species = True
outcome.analyses['13C NMR'].details = '101 MHz, CDCl3'
outcome.analyses['13C NMR'].data['peaks'].string_value = r'27.1, 38.4, 62.4, 127.7, 128.3, 129.6, 136.2, 176.3'
outcome.analyses['13C NMR'].data['peaks'].description = 'List of peaks'

outcome.analyses['thin film IR'].type = analysis.IR
outcome.analyses['thin film IR'].is_of_isolated_species = True
outcome.analyses['thin film IR'].details = 'neat film, NaCl'
outcome.analyses['thin film IR'].data['peaks'].string_value = r'3064, 3031, 2973, 2932, 2906, 2872, 1751, 1496, 1479, 1456, 1273, 1116, 1029, 738, 698'
outcome.analyses['thin film IR'].data['peaks'].description = 'List of peaks [cm-1]'

outcome.analyses['HRMS'].type = analysis.HRMS
outcome.analyses['HRMS'].is_of_isolated_species = True
outcome.analyses['HRMS'].details = 'ESI-TOF'
outcome.analyses['HRMS'].data['expected'].float_value = 298.1802
outcome.analyses['HRMS'].data['expected'].description = 'Expected m/z'
outcome.analyses['HRMS'].data['found'].float_value = 298.1794
outcome.analyses['HRMS'].data['found'].description = 'Observed m/z'

outcome.analyses['quantitative NMR'].type = analysis.NMR_1H
outcome.analyses['quantitative NMR'].is_of_isolated_species = True
outcome.analyses['quantitative NMR'].details = 'Quantitative NMR using 1,1,2,2-tetrachloroethane (>98%, purchased from Alfa Aesar) in CDCl3'

# A single product was desired and characterized
product = outcome.products.add()
product.identifiers.add(type='SMILES', value='O=C(C(C)(C)C)ON(CC1=CC=CC=C1)CC2=CC=CC=C2')
product.identifiers.add(type='NAME', value='N,N-Dibenzyl-O-pivaloylhydroxylamine')
product.mass.CopyFrom(unit_resolver.resolve('27.75 g'))

# Define which analyses were used for which aspects of characterization
product.measurements.add(type='IDENTITY', analysis_key='1H NMR')
product.measurements.add(type='IDENTITY', analysis_key='13C NMR')
product.measurements.add(type='IDENTITY', analysis_key='HRMS')
product.measurements.add(type='YIELD', analysis_key='isolated_weight', percentage=dict(value=93.5, precision=0.5))
product.measurements.add(type='PURITY', analysis_key='quantitative NMR', percentage=dict(value=99))

Updates look wonderful. Thanks, Connor!

# Define which analyses were used for which aspects of characterization
product.measurements.add(type='IDENTITY', analysis_key='1H NMR')
product.measurements.add(type='IDENTITY', analysis_key='13C NMR')
product.measurements.add(type='IDENTITY', analysis_key='HRMS')

Would just consider adding the IR analysis as an IDENTITY measurement as well:
product.measurements.add(type='IDENTITY', analysis_key='thin film IR')
Can look at notebooks soon.

@michaelmaser any additional comments? It would be great to merge this today.

Will take a look today. Sorry for delay, been a busy week.

Notebooks look great. Left some comments, but think anything could be addressed later if desired. Otherwise, LGTM.

@skearnes good to merge?

@skearnes good to merge?

Please resolve Mike's outstanding comments, and then we're good to go! Thanks for pushing this forward. After the merge, I'll create a new release for ord-schema and start working on the migration for the other repos. I'll work on new branches from main so we can all contribute, like we did here.

PS it's fine with me if you want to merge now and address e.g. separating standards in subsequent PRs. This PR is getting huge and we can start working on the migration in the meantime...


⚠ ClamAV has degraded performance

In ea92dcd, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 52 ms

Resolved: ClamAV performance has improved in fe2bde2.


Lora data not being transmitted

@sighmon I have opended a new issue to keep this separate and hopefully not to confuse things.
In the rasp pi logs I am seeing the sensor readings taking place every 15 minutes and it indicates the sensor readings are being transferred over lora.
However on my lora wan gateway and the TTN console I am not seeing every message the rasp pi shows it is transmitting.  The logs below show fcnt 1996 and fcnt 1994 that are missing.  I have just shown these two messages missing but it is a regualr occurance with many messages not being received.
The link gives you access to the log files  https://drive.google.com/drive/folders/1cHFf9MOTQUQjSvfz3X5BP_q5q7KFV0ct?usp=sharing
pi_testlog - raspberry pi log file
packet_log..csv - my lora gateway
eui...json - ttn console end device live data log

@mauriceatkinson I've requested to view the log files, but in the meantime we could try re-joining before sending each time and see if that helps.
Try: git pull and see if anything changes.
But yes, the problem is with the dragino library, so you'd have to ask them to handle the error you're seeing. They may have some other suggestions of how to avoid it too.

@sighmon this change looks to be a backward step as now lots of packets are missing and no data is being forwarded by TTN via mqtt.  Could you reverse this change?
I have made contact with the dragino developer so lets see what they say.

@mauriceatkinson No problems - I've reverted that change now. So do another git pull. Fingers crossed the developers come back with a solution.

@sighmon thanks.  I contacted the developer and he found a problem so currently I am testing this out.  As I have got his attention I will raise the Duty Cycle issue as well.

Fixed via https://github.com/BNNorman/dragino-1/commit/16b17dd33c2e2b9e4b727c5ace4479214d460578


Changes to the overlay and "Best Deals On Smartphones.." and footer

these are the changes

Can you please check this.