Proper method for including external JS libraries?

(cc: @Carreau from our Twitter conversation)
Not sure if this is really IJulia or Jupyter, but it's for Vega.jl...
What's the proper way of including JavaScript libraries? Vega.jl has 5 JavaScript libraries; when I pop open a new webpage see code here, this obviously works because it's valid HTML.
What's the correct way to have these libraries available within in Jupyter Notebook, after using Vega gets called? I used to be able to have success with this code, just dumping the JS using readall. But this can't be the right way to do this....in fact, it doesn't work with the new version of the Vega libraries.
Any help is greatly appreciated!

Quickly, and I'll try to re-respond later is to drop a require dependencies in the nbextension/... directory, that if your library just use require(['nbextension/dep'], function(dep){...}) it should load at the right time, and avoid the need to re-init, the libs.
We are pondering (or is it already implemented ?) a per-kernel nbextension or folder when kernel can put js libraries.
I understant that it might need an extra non-obvious step to drop the libs in the right places. We are working on a way to have kernel be able to "expose" javascript in a better way, but we miss manpower as usual to work on that.

I'm not sure I'm following here, as other libraries seem to be able to plot inline in Jupyter Notebook without having anything in the nbextension as far as I know.
If I were to put the Vega/d3 libraries in the nbextension folder, are you saying that I could call that folder as it's own library of sorts and it would load the JavaScript? Ideally, the libraries would live in my package folder, so that I could update them as Vega gets updated, instead of having to write out to the nbextensions folder each time.
This blog post shows that that in Python, you can load JavaScript via a magic; does Julia have anything like this?

You can just use display("text/html", "....") to output HTML text (potentially including Javascript) to be inserted into the notebook, although this has the issues describe in jupyter/notebook#319.  @Carreau, is the IPython %javascript magic doing anything different from this?

Nop, that's exactly what it does. Here is the full source:
    @cell_magic
    def javascript(self, line, cell):
        """Run the cell block of Javascript code"""
        display(Javascript(cell))

Thanks @stevengj, that confirms what I was doing.
I have this code in my module; when calling using Vega, the 5 ")
  display("text/html", "")
  display("text/html", "")
  display("text/html", "")
  display("text/html", "")

Where I'm getting stuck is that I can see these script references in the DOM in the div.output_subarea.output_html.rendered_html div . Additionally, d3 gets created in the global scope. However, vg does not, which is what I actually need to reference in order to render Vega graphs. When I create a stand-alone HTML file, vg and d3 are both available at the JavaScript console.
So, if anyone could help me with this, I'd be greatly appreciated. The Vega folks already tried to help me, but I'm pretty convinced that this is just some general JavaScript problem. I don't know JavaScript, so I'm just trying everything and haven't been successful thus far :(

I finally have gotten over the hurdle, realizing the issue is with the use of requirejs (which, having no front-end dev experience confuses the heck out of me.
Anyone running into this in the future, and assuming Jupyter doesn't change, this is vaguely the answer. I still have some bugs to work out, but running this code will eventually plot JS charts inline in the Notebook.
import Base.writemime
function writemime(io::IO, ::MIME"text/html", v::VegaVisualization)

        spec = JSON.json(tojs(v))
        divid = "vg" * randstring(3)

        display("text/html", """

              """)
end

require.config({paths: {vega: "http://vega.github.io/vega/vega.min"}});

Boom, Man in the middle. -> https  please, or you have remote code execution. (edited your snipets).
And also some browser might just plainly refuse to load that in authenticated context.
you should be able to install vaga.min in nbextension and require nbextension/vega to have it working with no internet connexion.
Happy you figured it out.

Thanks for noticing the https issue @Carreau, a silly oversight on my part.
When you say "install in nbextension", is it literally dropping a folder ~/.ipython/nbextensions/vega, then using similar requirejs code to load the libraries? You hit on the main thing I still haven't accomplished, which is that I don't want to require Internet.

yes, if you install in ~/.ipython/nbextensions or ~/.jupyter/nbextensions or global /usr/local/jupyter.. etc path you can require nbextension/vega/vega.min. The exact folder I'm don't remember, but on 4.0 you coud shell out to jupyter --path [--json] to get the various location.
There is also a nbextension install command to which you can give a folder and it should copy/link it into the right place.

Great, thanks so much. Hopefully I can solve this today before the holiday weekend!

@randyzwitch is there still a problem with the solution you sketch out above with the order that external scripts are loaded.
Doesn't it need to be something like the following to ensure that dependancies between d3, vega, and vega-embed are loaded in the right order?
  require.config({
    paths: {
      d3 : '//d3js.org/d3.v3.min.js?noext',
      vg : '//vega.github.io/vega/vega.js?noext',
      vl: '//vega.github.io/vega-lite/vega-lite.js?noext',
      vg_embed : '//vega.github.io/vega-editor/vendor/vega-embed.js?noext'
    },
    shim: {
      vg_embed: { deps: ["vg", "vl"] },
      vl: {deps: ["vg"]},
      vg: {deps: ["d3"]}
    }
  })

Although having said that. I can't get the above to work either. I get a util not defined error, and I haven't figured it out yet. Any suggestions much appreciated.

Same issue mentioned here.

I don't use vega-embed, but here's how I render Vega inline for Jupyter Notebook.
https://github.com/johnmyleswhite/Vega.jl/blob/master/src/render.jl#L45-#L74
Not saying it's what a JavaScript developer might do in terms of elegance, but it has worked for many months now.

Thanks @randyzwitch. I also ended up getting around it by switching to an iFrame with srcdoc (which means you can just add the scripts tags as if it were a standalone HTML page), but that's also probably not particularly elegant.

Any chance there's an update for 2023 that makes this easier?


React Images v1 🎉

For the past few months I've been rewriting react-images from the ground up. Having solidified some of the core concepts, i'd really like to hear your thoughts 🙂

Check out the v1 branch -->

The main impetus for rewriting this library was providing consumers with the building blocks necessary to create their component. Need extra buttons in the header? No problem, import react-images' <Header /> component and enhance it. Need to render videos rather than images? No problem, replace the <View /> component and pass the data required to the views property.
General notes:

At the moment the v1 branch is just a create-react-app until we decide on a build setup
Much of the state is now internal, simplifying implementation for the consumer
This project now leverages the amazing react-view-pager, which takes care of the touch/swipe boilerplate via react-motion
A great deal of the component architecture has come from recent R&D with @JedWatson for version two of react-select

Features of v1:

🛠   Comprehensively typed (flow)
📱   Support for touch devices
📺   Fullscreen support on desktop devices
🖼   Carousel without modal dialog
🏖   No restrictions on data shape
🚀   Replaceable component architecture

Many of the requirements, issues, and feature requests should be addressed by the new customisable component architecture without requiring changes to the core.
What this means for v0.x
No need to worry. I, together with @neptunian (the primary maintainer, who's doing an amazing job!), will continue to release critical bug fixes. We won't however be implementing enhancements or new features unless provided in a rock-solid PR.
The new API is a massive departure from v0.x, so there won't be an "upgrade guide" as such. I'll endeavour to provide a handful of examples to cover most use-cases and welcome anyone else to do the same.
Still left to do:

[x] settle on and implement a build tool
[ ] unit tests for each component
[ ] integration tests for carousel/modal
[x] lockdown the API and release an alpha
[x] light-weight theming/styling for components
[x] add classnames for those who prefer styling with CSS
[x] default components implement ARIA best practices

I'd like to thank everyone who has contributed to this project over the years, without your support react-images wouldn't be where it is today. Cheers to the open source community 💖
Looking forward to your feedback!

It would benefit testing if v1 could be published to npm with a beta or alpha version number and the next tag. See https://docs.npmjs.com/getting-started/using-tags.

@jossmac Is there a way we can help test this? What is the recommended way to use the v1 branch instead of the npm package, as it stands today?
Super excited about this! 🎉

Update
Checkout the demo site: v1 Alpha Demo -->
Alpha releases of<EMAIL_ADDRESS>available on npm:
yarn add react-images@next

CC @jorrit @codeluggage

Hi @jossmac,
I'm not seeing the examples at the bottom of the page...
On 19 February 2018 at 10:44, Joss Mackison<EMAIL_ADDRESS>wrote:

Update
Checkout the demo site: v1 Alpha Demo --> http://bit.ly/react-images-v1
Alpha releases of<EMAIL_ADDRESS>available on npm:
yarn add react-images@next
CC @jorrit https://github.com/jorrit @codeluggage
https://github.com/codeluggage
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jossmac/react-images/issues/199#issuecomment-366559206,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABs6x80ldn0eJChOs_et_COUk8zXWz0Kks5tWLXcgaJpZM4RvOFb
.

Works ok in Chromium on Linux but not FF. Quick inspection shows that each
thumb in the gallery with class .css-1vnvkj1 has a height of zero.
On 19 February 2018 at 10:49, Ben Howell<EMAIL_ADDRESS>wrote:

Hi @jossmac,
I'm not seeing the examples at the bottom of the page...
On 19 February 2018 at 10:44, Joss Mackison<EMAIL_ADDRESS>wrote:

Update
Checkout the demo site: v1 Alpha Demo --> http://bit.ly/react-images-v1
Alpha releases of<EMAIL_ADDRESS>available on npm:
yarn add react-images@next
CC @jorrit https://github.com/jorrit @codeluggage
https://github.com/codeluggage
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jossmac/react-images/issues/199#issuecomment-366559206,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABs6x80ldn0eJChOs_et_COUk8zXWz0Kks5tWLXcgaJpZM4RvOFb
.

Not proposing this as a solution but I've just hacked it so it works in FF
by removing position:absolute; from the image class .css-3c3ju2 (might give
you an idea at least)!
On 19 February 2018 at 10:53, Ben Howell<EMAIL_ADDRESS>wrote:

Works ok in Chromium on Linux but not FF. Quick inspection shows that each
thumb in the gallery with class .css-1vnvkj1 has a height of zero.
On 19 February 2018 at 10:49, Ben Howell<EMAIL_ADDRESS>wrote:

Hi @jossmac,
I'm not seeing the examples at the bottom of the page...
On 19 February 2018 at 10:44, Joss Mackison<EMAIL_ADDRESS>wrote:

Update
Checkout the demo site: v1 Alpha Demo -->
http://bit.ly/react-images-v1
Alpha releases of<EMAIL_ADDRESS>available on npm:
yarn add react-images@next
CC @jorrit https://github.com/jorrit @codeluggage
https://github.com/codeluggage
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jossmac/react-images/issues/199#issuecomment-366559206,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABs6x80ldn0eJChOs_et_COUk8zXWz0Kks5tWLXcgaJpZM4RvOFb
.

@benhowell cheers mate, fixed now! glam and flex weren't playing nice.
Also added prop type documentation 💃

Wha about this?
const images = [{ src: 'path/to/image-1.jpg', src: 'path/to/image-2.jpg' }];
You define array of images in doc in this way.

yarn add react-images@next is currently failing for me 😩
Module not found: Can't resolve 'react-images' in '...'

Looking at node_modules/react-images/package.json I got
{
  "version": "1.0.0-alpha.1",
  "main": "lib/index.js",
  "jsnext:main": "dist/react-images.es.js",
  "module": "dist/react-images.es.js",
}

There is no lib or dist directory, only src is there.. maybe something is missing on the publish script?

@weslleyaraujo I'm having the same issue.

@codeluggage @damienbrz @weslleyaraujo the files should be on NPM now

@neptunian should the version be 1.0.0-alpha.2 instead of 1.0.1-alpha.1? 🤔

@kachkaev i was wondering that myself

@kachkaev you're right, i'll fix that

@neptunian Unfortunately, the lib folder is still not published in 1.0.0-alpha.2.

@neptunian just installed the alpha version and the lib folder is not there.

@ArVan @kripod please use yarn add react-images@next

1.0.0-alpha.3 working great for me 👍

Unfortunately, SSR is still not working (see #247 for further details).

Ran into the SSR issue with Netlify as well.
The stable version has a nice little canUseDom.

@jossmac @neptunian Will the SSR issue be fixed any time soon? It would be great if v1 could be released with SSR support.

@jossmac @neptunian Will the SSR issue be fixed any time soon? It would be great if v1 could be released with SSR support.

I think that right now no one work on this library and the best option is to release v1 and then add improvements .

Is there any plan to released this version ? I feel like nobody is taking care of this library anymore.
You right, no one cares about this library.
@jossmac @neptunian  If you don't want to support it anymore so give the keys to someone else that want to that.

@OriAmir There hasn't exactly been any PRs to fix any issues with this so it wouldn't make much of a difference.  If someone would like make a PR for the SSR issue, that would be welcome.  Otherwise I'll have to get it to when I get a chance which should be soonish!

@OriAmir There hasn't exactly been any PRs to fix any issues with this so it wouldn't make much of a difference. If someone would like make a PR for the SSR issue, that would be welcome. Otherwise I'll have to get it to when I get a chance which should be soonish!

@neptunian
I think that the best option is firstly to release v1 and then fix the SSR and other issues because the library needs some refresh -there are many issues without replay and any new release not happened in the last 6 months.

@oriamir the SSR issue affects a lot of people who might use this library, considering how popular frameworks like Gatsby are becoming.
If you want the current version released as a V1, what’s stopping you from using the alpha? Changing it from alpha to V1 doesn’t make any functional difference.
Give the maintainers time to do their thing and live their lives. The great thing about this project being open-source is that nothing is stopping you from submitting a PR that fixes what you perceive to be an issue.
@neptunian, thanks for the work you’ve put into this.

if you install the latest next version (react-images@next), the SSR app should be fixed.

I want to mention before the release of v1 that I do not plan to maintain v0.x.  I can create a branch for it, if anyone else wants to work on it, and accept PRs there.

@neptunian  Thank you for the amazing work! And everyone here as well


compress css and js for production

hi
Is it possible to reinstate the "gulp dist" command or is there an alternative method for compressing my CSS/JS files for production?

Hi there, the dist task included in codyframe v3 works regardless of the framework, so you can use the same for codyframe v4.

In the latest version of CodyFrame, is it ok to change the order of utility imports in the style.scss file?
Original order:
@use 'typography';
@use 'icons';
@use 'buttons';
@use 'forms';
@use 'components/**/*.scss';
@use '../../node_modules/codyframe/main/scss/util';

Modified order:
@use '../../node_modules/codyframe/main/scss/util';
@use 'typography';
@use 'icons';
@use 'buttons';
@use 'forms';
@use 'components/**/*.scss';

Otherwise the /*! purgecss start ignore / and /! purgecss end ignore */ comments somehow encompass all components and styles, and purgeCSS will become ineffective.

Hi there, you cannot modify the order, utilities should be imported at the end.
As for the purge, you should not use it for the components. Some of the component classes are used in JS (defined dynamically) so purge may break them. This is the reason why we add the purgecss start/purge end comments, to make sure only utility classes are properly cleaned. This is the same logic we used in CodyFrame v3.
Please also note that the components only have the classes required for them to work properly, so there should be no need to clean them with purgeCSS.


Provisions Linux test executing instances

Description
This PR performs the provisioning of the test executing VM instances by installing the Infrastructure Agent (in its latest released version), Fluent Bit (with the version pinned for each OS distro/version and arch) and NodeJS (will be used to execute the test suite later). The corresponding provisioning of Windows VMs will be done in a separate PR.
I will provide some explanatory comments to ease the PR review.
Testing
The PR has been tested locally by using the "make local" target and running the Ansible playbooks against real EC2 instances.


Perhaps wrong license in LoRa examples

I was taking a look at the new LoRa examples and saw that the LICENSE.txt file is the GFDL v3, while the README says that the examples are GPLv3 because they're derived from a GPLv3 GNU Radio OOT module (which makes sense). So it seems that the wrong file was added as LICENSE.txt.

Arg, sorry. I fixed it.


Add in "Generalized" States the README Talks about

Need to add in some of the Generalized States mentioned in the README. I got... excited and pushed the new version with a README that was new, and I didn't fully vet what I was writing.

not needed afterall.


Cannot read properties of undefined (reading 'startsWith')

Environment

Operating System: Linux
Node Version:     v20.6.1
Nuxt Version:     3.7.3
CLI Version:      3.9.0
Nitro Version:    2.6.3
Package Manager<EMAIL_ADDRESS>
Reproduction
import { useCompression } from 'h3-compression';

export default defineNitroPlugin((nitroApp) => {
    nitroApp.hooks.hook('render:response', async (response, { event }) => {
        if (!response.headers?.['content-type'].startsWith('text/html')) {  // undefined response.headers?.['content-type']
            return;
        }

        await useCompression(event, response);
    });
});

Describe the bug

I not see it any more, may be that case is registered when some errors is in nuxt config.

Cannot read properties of undefined (reading 'startsWith')
at  (./server/plugins/compression.ts:9:1)
at ./node_modules/hookable/dist/index.mjs:48:66
at ./node_modules/hookable/dist/index.mjs:48:56
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at  (./node_modules/nitropack/dist/runtime/renderer.mjs:29:5)
at async Object.handler (./node_modules/h3/dist/index.mjs:1630:19)
at async Server.toNodeHandle (./node_modules/h3/dist/index.mjs:1840:7)
Additional context
No response
Logs
No response

You can add an extra nullish coalescing check after array lookups:
if (!response.headers?.['content-type']?.startsWith('text/html')) {


Master Issue: Enable try support in Fuzzer

Description
Running fuzzer with -retry_with_try brings up a bunch of new fuzzer bugs. This issue will detail all of these bugs.
Error Reproduction
No response
Relevant logs
No response

2. Regexp functions do not handle Try correctly.
Functions : "regexp_replace",
"re2_extract_all",
"re2_extract",
"re2_search",
"re2_match",
"like"
do not handle Try correctly and thus fail Try fuzzer.

3. Common and Simple eval dont match when executing expression distinct_from(276619011,clamp(codepoint("c0"),round(null),floor(807223209)))
Input: [DICTIONARY VARCHAR: 100 elements, 9 nulls], [CONSTANT VARCHAR: 100 elements, {sKT-dKablwem]q.wWEn4$W)Rh?h&lWxNA<fW"q_'c@;_\S4]V;=-f|>3/m)(i}v0n]
Stack Trace:
 E20230104 15:09:01.423604 8106041 Exceptions.h:68] Line: /Users/kpai/src/Velox/velox/expression/tests/ExpressionVerifier.cpp:67, Function:compareVectors, Expression: left->equalValueAt(right.get(), i, i) Different results at idx '17': 'null' vs. '1', Source: RUNTIME, ErrorCode: INVALID_STATE
libc++abi: terminating with uncaught exception of type facebook::velox::VeloxRuntimeError: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Different results at idx '17': 'null' vs. '1'
Retriable: False
Expression: left->equalValueAt(right.get(), i, i)
Function: compareVectors
File: /Users/kpai/src/Velox/velox/expression/tests/ExpressionVerifier.cpp
Line: 67
Stack trace:
# 0  ..

I will have PR for this soon.

4. Infinite loop in date_parse .
With skipFunctions: "regexp_replace",
"re2_extract_all",
"re2_extract",
"re2_search",
"re2_match",
"like",
"regexp_extract",
And running fuzzer as : --seed<PHONE_NUMBER> --duration_sec 3600 --logtostderr=1 --minloglevel=0 --retry_with_try

@kgpai Krishna, do we have separate fuzzer-found issues for regex and date_parse functions? If not, let's create these.

Try support has been added and has been stable in fuzzer for somewhile so closing this issue.


[WIP] update CHANGELOG for 17.03.0-ce

ping @aaronlehmann @mavenugo @aboch @thaJeztah

We need to mention and explain the versioning change.
Here are changelog entries for swarm mode changes:

Fix possibility of tasks getting stuck in the "NEW" state during a leader failover https://github.com/docker/swarmkit/pull/1938
Fix extraneous task creations for global services that led to confusing replica counts in docker service ls https://github.com/docker/swarmkit/pull/1957
Fix problem that made rolling updates slow when task-history-limit was set to 1 https://github.com/docker/swarmkit/pull/1948
Restart tasks elsewhere, if appropriate, when they are shut down as a result of nodes no longer satisfying constraints https://github.com/docker/swarmkit/pull/1958

And another networking one:

Fix port allocation when multiple published ports map to the same target port https://github.com/docker/swarmkit/pull/1835

@aaronlehmann @mavenugo @thaJeztah PTAL

ping @friism

(not maintainer) lgtm!

LGTM

LGTM

Hi. Where can I read up on the new versioning scheme?


Command line flags do not work after code formatter/beautifier

Description

Visual studio 2017 automatically formats code when I paste code from github.com/doctest.h. Everything works fine, but a few command line flags.
I guess, additional spaces in calls DOCTEST_PARSE_AS_BOOL_OR_FLAG do the bug.
Steps to reproduce

Add spaces (around '-') to:
DOCTEST_PARSE_AS_BOOL_OR_FLAG(dt - exit, dt - e, exit, false);
Now, shouldExit() always returns false (with "--exit" arg).
 This isn't a doctest issue per-se.
Well I'm not sure what to do about this... Can you download the file and just open/use it instead of pasting its contents in VS?

Sure, I'v already downloaded the file (and exclude it from pre-git-hook style tools).
I think this format-depended code a bit fragile, so it would be nice if a compilation error occurred in case you accidentally reformat sources.

I'll think about how to avoid this, thanks :)

BTW is there some penalty (compile time) with quoted strings as macros arguments?

If you are wondering if I can pass something like "dt-success" to DOCTEST_PARSE_AS_BOOL_OR_FLAG and just concatenate in the macro the given string with "=" in here - I'm wondering the same - and will try it :)


Error in Line 8

Hi there. Running into the following error after executing:
rmc -t markdown ffb3a170-cf8e-49f0-a4a9-ba4e8b52c840.rm
Error is:
Traceback (most recent call last):   File "/home/parallels/.local/bin/rmc", line 8, in <module>     sys.exit(cli()) TypeError: command.<locals>.decorator() missing 1 required positional argument: 'f'
I'm running Ubuntu 20.04.2 LTS. As I'm running a VM on an M1 MacBook, it's ARM architecture (shouldn't matter, but figured I would mention it).
Thank you for the open source work!

Hi,
I also got the error with python 3.10, click 8.0.3 and rmc 0.1.1
It is related to https://github.com/pallets/click/issues/1359
@click.command() should have parentheses like any function call (in /usr/local/lib/python3.10/dist-packages/rmc/cli.py).
Regards,
Pascal Noisette


Cannot fetch API keys for other users

I caanot fetch API keys for other users as I am getting  Error: The specified IAM ID does not match the IAM ID in the token., Code: 400. This happens when I run the following code:
service_client = IamIdentityV1.new_instance()
user_api_keys_response = service_client.list_api_keys(account_id=account_id, iam_id=iam_id, include_history=True).get_result()

I presume my error is suggesting that you cannot fetch API keys for users other than the user you are requesting with, as I discovered from the docs which state: ‘If a service IAM ID is specified in iam_id then account_id must match the account of the IAM ID. If a user IAM ID is specified in iam_id then account_id must match the account of the Authorization token.’.
Is there a way to fetch API keys for other users?
''

Only users can fetch their own user API key. Admins however can list the API keys of the account. So basically the above code should work if you don't specify the iam_id parameter but specify scope=account. You can additionally specify type=user to reduce the result to user API keys only.

Ok, thank you for the clarification.


Update for ipywidgets 6.0 beta

@jasongrout

Had to add yaml to the the testing miniconda...

Seems to work well. I have an issue with the tooltip graph in the second example of the Interaction/Mark Interaction example. The tooltip graph doesn't show up, just a transparent black rectangle. But it seems to do that on master too, so I'll merge this and we can debug it as a separate issue.

@jasongrout The tooltip issue seems to be coming from ipywidgets Button doesn't display font-awesome icons. Do you know why that could be happening?


Support implementing functions in Python

Resolves #230

lgtm @jordemort

Thanks, this looks pretty good. Two things:

The build fails on Python 3.7, 3.8, 3.9 because the exception message is not exactly what the test expects. I need to update my Python versions here and drop 3.7 because it's EOL, but 3.8 and 3.9 are still supported, so please loosen up the test so that it works there.
I'm not sure exactly where this capability should be mentioned in the docs, but it should be mentioned somewhere. Unfortunately, the bulk of the documentation is embedded in docstrings in starlark.c in a very ugly way.

@jordemort How's that?

This is looking good; the macOS builds are broken through no fault of yours. I fixed that, and dropped Python 3.7, and promoted Python 3.12 to production, and updated a few other dependencies. Would you mind rebasing on to / merging in my latest main? After that, if all the tests are passing, I think I am ready to merge it.

@jordemort Should be good to go!

@jordemort any updates?

@brandonchinn178 I've merged it! There's a few other things I want to get to before doing another release, though, and I'm not sure when I'm going to get to them; if you or anyone else wants to run with these before I am able to make time, I am happy to review and merge the results:

It looks like I need to enable 2FA on test.pypi before it will let me do test uploads again
The embedded version of Starlark needs to be updated, but it's going to necessitate an API break, re: https://github.com/caketop/python-starlark-go/pull/204
There's some open issues related to adding support for structs, json functions, and other loadable stuff that I would like to include in a 2.0, if I'm going to have to break the API anyway

Thanks @jordemort! Is there a reason why starlark-go "has" to be updated? I don't see why function support can't be released now in the 1.x series, decoupled from implementing all those 2.x features.
The only reason I'm asking is because it'd be nice to have a prebuilt wheel with function support, so I don't have to require users of my script to have golang installed to install python-starlark-go

@brandonchinn178 I'll see if I can get an interim release out for you, I just feel bad pushing one with such an out-of-date release of starlark-go. Hopefully I'll get some time later in the week to chase the bats out of the CI.

I'd really appreciate that! We don't mind the old version of starlark-go. Thanks a ton!


Sync FAQ sections on site

There are couple different FAQ sections on the site:

Built: http://numenta.com/business-strategy-and-ip/#faq

Source: https://github.com/numenta/numenta-web/blob/master/numenta.com/pages/business-strategy-and-ip/_Section.jsx#L258-L314

This one uses HTML definition lists (<dl>, <dt>, <dd>)

Built: http://numenta.com/htm-studio/#faq

Source: https://github.com/numenta/numenta-web/blob/master/numenta.com/pages/htm-studio/index.jsx#L128-L141

This one is just <div>s.  Visually, on desktop widths, there is unwanted padding on the left of each FAQ answer/definition.

These should match in style, markup, etc.  Maybe some refactoring.
See related issue: https://github.com/numenta/numenta-web/issues/15
cc @jsfowles

@brev - Can I work on this now or does #15 need to be completed first?

This can be done separately first, although I probably need to define it better, let me get back to you on requirements.

ok @jsfowles go ahead

meh.


Add TryFrom<&Vec> and TryFrom<&Array>

The existing TryFrom<Vec> and TryFrom<Array> only need references so this just changes the original signatures to use references and adds new wrapper implementations for TryFrom<Vec> and TryFrom<Array>.
This builds on top of #356 (although it can be implemented without that change) so all of the changes from #356 are included in this diff as well. I don't know how to prevent that but anyway, don't merge this unless #356 has been merged.

@edlanglois We should implement TryFrom instead of TryInto. Says by std doc, TryFrom implies TryInto but it does not in vice versa. A similar fix can be found in my old #297 PR.


Feature: --no-cache takes a regexp to ignore matching lines in Dockerfile

modifies CLI and server to accept a regexp for no-cache
adds defaultable flags to mflag to support existing bool no-cache
applies to matching RUN and ADD lines

Closes #1996

What ever became of this idea?

This proposal will solve this issue, and in much more elegant way, IMO. If what folks are looking for is a way to invalidate cache when something like a git repo actually changes you could take the following approach:
$>docker build --build-arg git_commit=$(git rev-parse HEAD) ...
//the file:
FROM busybox
RUN git clone https://git.foo.com/myrepo.git
ARG git_commit
RUN git fetch https://git.foo.com/myrepo.git $git_commit
RUN git merge $git_commit

The proposed patch has been merged or approved, I believe, so hopefully we'll see it out soon.

in the more generic case you can also do:
$ docker build --build-arg bust=$(date) ...

FROM busybox
...
RUN echo $bust
...

will never cache from that RUN line thru the rest of the Dockerfile.  Just too bad that people need the hack in the first place.


Dev task - Fetch staff function shouldn't need to take in URL

Minor thing, but I feel the fetchStaff function should have the URL without it being a parameter. As it stands, it is just a fetch+auth function

The function definitely needs renaming, because as you pointed out it can do much more than just fetch staff (can also be used for engagements for example).
We don't want to end up in a situation where we have a bunch of functions that are identical except for the URL (fetchStaff, fetchEngagements and also addStaff and removeStaff, which return an updated staff list).
What we could do is :curry: the current function, so change from:
const fetchStaff = (token: string, url: string): AppThunk => (
to
const fetchStuff = (url: string) => (token: string): AppThunk => (
and then create a bunch of children functions:
const fetchStaff = fetchStuff(STAFF_FETCH_URL);
Would that be acceptable?

One possible solution would be to create a generic 'apiService' class which has a 'get' method on it, which takes a url (which should be a const somewhere), and appends any headers required. This will also make it easier to append a token to every single request later on.

You could still refer to 'fetchStaff' when you want to specifically fetch the staff, and call a apiService.get() method to which you pass a specific getStaff url, with any specific getStaff args

I think, Ideally the react components shouldn't see the URL.  If we had functions for each endpoint, those function job could be to know the URL and then call the actual getter that adds auth?


Improve cross references for declaration merging

Currently, a cross reference can only ever point to a single element. For features such as declaration merging (i.e. with interfaces in TypeScript), it might make sense to adapt the cross reference resolution/grammar to allow multiple elements for a cross reference target, so that these can be merged during the scoping process.

Ideally, we would declare explicitly where we want to resolve to multiple targets in the grammar. At the declaration site, it could look like this:
interface Foo {
    myRef: @Target[]
}

But how do we specify it at the assignment? Maybe with a *?
Foo: myRef=[Target*: ID]


The NPM install for royale-js-swf does not allow -y, --yes or other ways to confirm automatically

In order to use royale-js-swf on a build or continuous integration server, it is necessary to be able to say "yes" to the install prompt automatically. I believe the "prompt" NPM library is already being used, so it would be great to add support for overriding:
https://www.npmjs.com/package/prompt#skipping-prompts
Using yes | npm install on a Linux install works, but returns an error code due to the way it interacts with NPM, so that also breaks our builds.
Thank you :smile:

The download dependencies code runs as a postInstall script during the npm install lifecycle.  I am not sure if we can pass params to an npm install call.
This ticket: https://github.com/npm/npm/issues/8431 was closed a a wont fix :-(
Would setting up env variables be a valid option for you?

Using an environment variable may be the best technical solution to this at the moment, it's a pity that argv isn't passed forward. It might make sense to still add the argv override method, in case a -y could be forwarded in the future, or if someone is manually running npm run postinstall later. The key here is we could install in a headless way for continuous integration testing or our build server :smile:
Thank you so much for your help!

Understood.  I will try to get this one working soon and we can test it.

Added check to see if ACCEPT_ALL_ROYALE_LICENSES env var is set.  Let's track the prompt -y option as a separate issue if needed.  I believe this should satisfy the automated build usecase.


Rename LazyListScope.TopicBody LazyListScope.topicBody

It appears that other extensions for LazyListScope start with a lowercase letter, but TopicBody starts with an uppercase letter. Therefore, I have changed it to start with a lowercase letter.

extensions which start with lowercase are not Composables.
Composables emitting UI elements should start with uppercase letter

It is better to rename userNewsResourceCard to UserNewsResourceCard


Pre commit ci

@anki-code it is better to create PR to add changes to the repo.
@gforsyth if possible, could you lock direct merge to the main branch ?

@gforsyth if possible, could you lock direct merge to the main branch ?

done -- also added maintain permissions for this repo to xore (which should have been there already)


Compile error in android studio when imported the sample projects

I imported BasicSample project. When i build it i get the following errors.
Is there any configuration issue with my android studio? Android studio version is 2.3.3
:app:compileDebugJavaWithJavac
E:\Android\experiments\android-architecture-components-master\BasicSample\app\src\main\java\com\example\android\persistence\ProductFragment.java:29: error: package com.example.android.persistence.databinding does not exist
import com.example.android.persistence.databinding.ProductFragmentBinding;
^
E:\Android\experiments\android-architecture-components-master\BasicSample\app\src\main\java\com\example\android\persistence\ProductFragment.java:43: error: cannot find symbol
private ProductFragmentBinding mBinding;
^
symbol:   class ProductFragmentBinding
location: class ProductFragment
E:\Android\experiments\android-architecture-components-master\BasicSample\app\src\main\java\com\example\android\persistence\ui\CommentAdapter.java:26: error: package com.example.android.persistence.databinding does not exist
import com.example.android.persistence.databinding.CommentItemBinding;
^
E:\Android\experiments\android-architecture-components-master\BasicSample\app\src\main\java\com\example\android\persistence\ui\CommentAdapter.java:104: error: cannot find symbol
final CommentItemBinding binding;
^
symbol:   class CommentItemBinding
location: class CommentViewHolder
E:\Android\experiments\android-architecture-components-master\BasicSample\app\src\main\java\com\example\android\persistence\ui\CommentAdapter.java:106: error: cannot find symbol
CommentViewHolder(CommentItemBinding binding) {
^
symbol:   class CommentItemBinding
location: class CommentViewHolder
E:\Android\experiments\android-architecture-components-master\BasicSample\app\src\main\java\com\example\android\persistence\ProductListFragment.java:30: error: package com.example.android.persistence.databinding does not exist
import com.example.android.persistence.databinding.ListFragmentBinding;
^
E:\Android\experiments\android-architecture-components-master\BasicSample\app\src\main\java\com\example\android\persistence\ui\ProductAdapter.java:26: error: package com.example.android.persistence.databinding does not exist
import com.example.android.persistence.databinding.ProductItemBinding;
^
E:\Android\experiments\android-architecture-components-master\BasicSample\app\src\main\java\com\example\android\persistence\ProductListFragment.java:45: error: cannot find symbol
private ListFragmentBinding mBinding;
^
symbol:   class ListFragmentBinding
location: class ProductListFragment
E:\Android\experiments\android-architecture-components-master\BasicSample\app\src\main\java\com\example\android\persistence\ui\ProductAdapter.java:103: error: cannot find symbol
final ProductItemBinding binding;
^
symbol:   class ProductItemBinding
location: class ProductViewHolder
E:\Android\experiments\android-architecture-components-master\BasicSample\app\src\main\java\com\example\android\persistence\ui\ProductAdapter.java:105: error: cannot find symbol
public ProductViewHolder(ProductItemBinding binding) {
^
symbol:   class ProductItemBinding
location: class ProductViewHolder
error: Cannot figure out how to save this field into database. You can consider adding a type converter for it.
error: Cannot find getter for field.
error: Cannot find getter for field.
error: Cannot find setter for field.
error: Cannot find setter for field.
error: Cannot figure out how to save this field into database. You can consider adding a type converter for it.
error: Cannot find getter for field.
error: Cannot find getter for field.
error: Cannot find setter for field.
error: Cannot find setter for field.
error: Cannot figure out how to read this field from a cursor.
error: Cannot find getter for field.
error: Cannot find getter for field.
error: Cannot find setter for field.
error: Cannot find setter for field.
error: Cannot figure out how to read this field from a cursor.
error: Cannot find getter for field.
error: Cannot find getter for field.
error: Cannot find setter for field.
error: Cannot find setter for field.
E:\Android\experiments\android-architecture-components-master\BasicSample\app\src\main\java\com\example\android\persistence\db\AppDatabase.java:31: warning: Schema export directory is not provided to the annotation processor so we cannot export the schema. You can either provide room.schemaLocation annotation processor argument OR set exportSchema to false.
public abstract class AppDatabase extends RoomDatabase {
^
30 errors
1 warning
FAILED
FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':app:compileDebugJavaWithJavac'.

Compilation failed; see the compiler error output for details.

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED in 6s
34 actionable tasks: 29 executed, 5 avoided (15%)

I am facing the same issue , currently I using "1.0.0-beta1" version for all architectural component.
@krishnaswamyna can you plz help.

@biswanathgit Few things to check, make sure the support library version, compile sdk version, build tools version are all the same . eg: all should start with 25.
I also downloaded and installed version 25 sdk. With this it worked for me. Problem for me was for some reason the android.jar in the sdk\platforms\android-25 was not proper

@krishnaswamyna here is my repo  after once you make or build you got that error. I have tried that you mentioned. But get that same error. If you checked it once. It will very helpful to me.

@biswanathgit android version i compiled ia 25. And i see that your version is 26.
Gradle i used is classpath 'com.android.tools.build:gradle:2.3.0'
I cannot actually compile it now as i do not want to update gradle or studio right now in middle of a project. So you could try with version 25 if it helps.


Unable to install azure-mgmt-compute 3.0.0 on python3.5

Trying to install azure-mgmt-compute 3.0.0 on python3.5 and getting the below error. I get that this is old version but we need this.
.2-py3-none-any.whl
Collecting cryptography>=1.1.0 (from adal<2.0.0,>=0.5.0->msrestazure~=0.4.11->azure-mgmt-compute==3.0.0)
Using cached https://files.pythonhosted.org/packages/15/d9/c679e9eda76bfc0d60c9d7a4084ca52d0631d9f24ef04f818012f6d1282e/cryptography-40.0.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 20, in
File "/tmp/pip-build-8b6lvfd4/cryptography/setup.py", line 90
print(f"    Python: {'.'.join(str(v) for v in sys.version_info[:3])}")
^
SyntaxError: invalid syntax
----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-8b6lvfd4/cryptography
You are using pip version 7.1.2, however version 23.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[root@ktserver4 rpms]#

Thanks for reaching out.
Unfortunately, azure-mgmt-compute 3.0.0 is deprecated and we recommend to use newer versions.
If you don't mind, could you share the reason why you want to use this old version?
Thank you.

Thanks @xiangyan99.
Yes we get that this is very old and unfortunately all our legacy code has been written with this version.  It would be very difficult for us, to change all of that in one go.  So if we have some workaround , till we re-write using new version it would help.

Thanks for the information.
New versions of cryptography does not support py3.5.
Can you try install cryptography<4 before you start installing azure-mgmt-compute 3.0.0?

We can close this , after installing all the dependencies which are supported , it works now.


refactor: upgrade project

:tada: This PR is included in version 0.89.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Skip cars that don't have vin

Fixes #133

I think that makes perfect sense.  Since we can't process information without vin anyway


National Emission Inventory

[x] Creation of CSV File
[x] Creation of MCF File
[x] Creation of TMCF File
[x] Unit Testing
[x] Code Linting
[x] DC Validation

@shifucun @Fructokinase please take a look - thanks!


It may better to remove #if DEBUG

Basically large APP will use multiple scheme and use different macro to distinguish, most of them may not in debug mode, and also we don't Introduce debug tool in release from pods, so it's no need to add #if DEBUG in func startLayoutInspection()

@miku1958 yes, mentioned scenario makes sense, however:

large app will use multiple schemes but opposite is also true - there are small apps which don't have such separation;
same applies for pods structuring to exclude debug tool from release.Again, not all apps have it :(
Current approach keeps LayoutInspector's API consumers in safe and requires minimal effort to onboard it into the project

@isavinskyi that's true... how about tell other others it's necessary to add Debug macro in readme?

@miku1958 DEBUG macro already mentioned in Readme here


Add a handful of integration tests

https://github.com/FriendsOfPHP/PHP-CS-Fixer/pull/3444 got merged, I merged 2.10 into your PR.
let us see...

@keradus It looks like the appveyor failures are unrelated to my changes.

@BackEndTea please note that I've resolved the merge conflict, so please update your local branch if needed

Let us not merge this one yet, we got no answer for class_attributes_separation,braces.test, and for that I want to manually test it

sure @keradus , I couldn't find a conflict but like hear your findings :)

maybe i won't find a thing, simply need to find an hour of calm and double check it

I found a prio issue after all and created the test.
Also updated the fixer factory for it, rebased, resolved conflicts and squashed.
I know people don't like early squashing but maintaining a PR for 10+ months causes this type of thing.

I know people don't like early squashing (...)

next time, simply keep single commit for original author's changes, and different commit for your changes - now, impossible to see what was your fix

I found a prio issue after all and created the test.
maintaining a PR for 10+

ha, I knew we had a prio issue somewhere here ! nice finding !

Thank you @BackEndTea.

Thank you @SpacePossum & @keradus


SNOW-876355: dataframe show throws out of memory

it seems dataframe show fetches rows to the client leading to out of memory errors for larger dataframes

@sonalgoyal -- Yes that's expected, .show() is an action and will fetch the entire DataFrame. If you're working with a dataset which exceeds the memory of the client, try using DataFrame.toLocalIterator() to incrementally fetch the results.
More info: https://docs.snowflake.com/en/developer-guide/snowpark/java/working-with-dataframes#performing-an-action-to-evaluate-a-dataframe

Thanks for the link @sfc-gh-jfreeberg. Though it is an action, the documentation mentions. printing 10 rows  and my understanding was to use it like a debug in the flow, just like Spark. In my opinion, if it is 10 rows, the entire result set should not be brought to the client but limited on the warehouse.


[8.0.13] enableSortingRemoval does not work as intended

Describe the bug
The enableSortingRemoval does not seem to work. AFAIU it this should be the inverted version of disableSortRemove from react-table v7. However as you can see in the example when clicking on the column header, the first time the sorting order changes to ascending. On the next click it changes to descending, but on the third click with enableSortingRemoval: false I would have expected to get back to ascending. Instead the column is stuck at descending on all follow-up clicks.
Your minimal, reproducible example
https://codesandbox.io/s/serverless-wildflower-khkcu9?file=/src/main.tsx
Steps to reproduce
The sample is equivalent to https://tanstack.com/table/v8/docs/examples/react/sorting + the option enableSortingRemoval: false.
Click on "Profile Progress" for at least three times. The behavior is the same for most columns, but for some strange reason the "Status" column seems to work as expected
Expected behavior
The sorting order switches between ascending and descending order with every click.
How often does this bug happen?
Every time
Screenshots or Videos
https://user-images.githubusercontent.com/5631865/176750185-363325f8-4777-4555-a188-44ef6478371c.mov
Platform
macOS, Chrome 103
react-table version
8.0.13
TypeScript version
4.7.4
Additional context
No response
Terms & Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I understand that if my bug cannot be reliable reproduced in a debuggable environment, it will probably not be fixed and this issue may even be closed.

Fixed in the latest release :)


Refactor components with React ES6

Is there any plan to refactor current components with react ES6 syntax? Thanks.

PR was merged, close the issue now.


Complete the dependencies and upgrade the interface to the new version.

What should be cleaned up or changed:
Complete the dependencies
The current Ianvs project has issues with unclear and incomplete dependencies, as pointed out in issues #103 and #106.
This poses significant challenges for users to configure the Ianvs environment.
I suggest:

Completing  the dependencies for Ianvs Core, and updating requirements.txt

Update outdated dependencies and align interface to new version
Meanwhile, some dependencies are severely outdated compared to the main versions, and certain APIs from older versions have been deprecated.  These outdated dependencies may cause confusion for users.
For example:

pandas is needed by core/storymanager/rank/rank.py with version 1.1.5 , while the latest version has reached 2.2.2 with many APIs deprecated
sedna has released 0.6.0, while the built-in package in Ianvs is 0.4.1

I suggest:

Updating outdated packages and upgrading the interface to the new version. (e.g. pandas interface in rank.py)

Remove examples/resources/
There are also some dependencies within projects under examples/.
Since these projects are not closely related to Ianvs Core, why should their dependencies be placed in examples/resources?
We can provide an intuitive requirements.txt for each example, which aligns with Python usage habits and helps reduce the introduction of large binary packages into the Git repository.
I suggest:

Remove the method of managing third-party dependencies through examples/resources/, and embed dependencies of examples/xxxxx into a requirements.txt within each example directory.

Why is this needed:
Improving dependencies and aligning the new version interface will help Ianvs solve complex legacy dependency issues, allowing users to quickly get started with Ianvs.

Considering that the API of dependencies in different versions of Python may vary significantly, we need to determine the minimum Python version supported by Ianvs.
Experiment on Quick Start - The PCB-AoI Example
I followed the Quick Start guide to run the PCB-AoI example. However, during the attempt, I found that this example currently has an irreconcilable dependency conflict.
I will provide the results of dependency configuration for different Python version environments:
Python 3.6
Step 1. Ianvs Preparation
We first need to install the sedna-0.4.1-py3-none-any.whl package located in examples/resources/third_party/.
In this step, we need to use the following command to complete the dependency installation:
pip install ./examples/resources/third_party/*
pip install -r requirements.txt

However, Sedna-0.4.1 specifies a dependency on joblib~=1.2.0, which requires Python>=3.7, so it cannot be installed and the sedna-0.4.1-py3-none-any.whl must be updated.
$ pip install examples/resources/third_party/sedna-0.4.1-py3-none-any.whl
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Processing ./examples/resources/third_party/sedna-0.4.1-py3-none-any.whl
Collecting six~=1.15.0
  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting PyYAML
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/62/2a/df7727c52e151f9e7b852d7d1580c37bd9e39b2f29568f0f81b29ed0abc2/PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB)
     |████████████████████████████████| 677 kB 3.2 MB/s
Collecting setuptools~=54.2.0
  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/9e/d4/b99a960314121a003e9f39c61dfde01a1010bb47661e193a7722f7f32d52/setuptools-54.2.0-py3-none-any.whl (785 kB)
ERROR: Could not find a version that satisfies the requirement joblib~=1.2.0 (from sedna) (from versions: 0.3.2d.dev, 0.3.2e.dev, 0.3.2f.dev, 0.3.2g.dev, 0.7.0d, 0.1a0.dev0, 0.2a0.dev0, 0.3a0.dev0, 0.3.1a0.dev0, 0.3.2.dev0, 0.3.2a0.dev0, 0.3.2b0.dev0, 0.3.2rc0.dev0, 0.3.3a0.dev0, 0.3.3b0.dev0, 0.3.3rc0.dev0, 0.3.4.dev0, 0.3.5.dev0, 0.3.6.dev0, 0.3.7.dev0, 0.4.0.dev0, 0.4.1.dev0, 0.4.2.dev0, 0.4.3.dev0, 0.4.4.dev0, 0.4.5.dev0, 0.4.6.dev0, 0.5.0.dev0, 0.5.0a0.dev0, 0.5.1.dev0, 0.5.2.dev0, 0.5.3.dev0, 0.5.4.dev0, 0.5.5.dev0, 0.5.6.dev0, 0.5.7.dev0, 0.5.7a0.dev0, 0.5.7b0.dev0, 0.5.7, 0.6.0a0, 0.6.0b0, 0.6.0b2, 0.6.0b3, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.7.0a0, 0.7.0b0, 0.7.0rc0, 0.7.1, 0.8.0a0, 0.8.0a2, 0.8.0a3, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.3.post1, 0.8.4, 0.9.0b2, 0.9.0b3, 0.9.0b4, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.10.0, 0.10.2, 0.10.3, 0.11a3, 0.11, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.12.4, 0.12.5, 0.13.0, 0.13.1, 0.13.2, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.16.0, 0.17.0, 1.0.0, 1.0.1, 1.1.0a0, 1.1.0, 1.1.1)
ERROR: No matching distribution found for joblib~=1.2.0

Step 2. Dataset and Model Preparation
Skiped because Step 1 Failed.
Python 3.7
Step 1. Ianvs Preparation
Since Python 3.7 is compatible with joblib~=1.2.0, no related issues were encountered. This step was completed successfully.
Step 2. Dataset and Model Preparation
In this step, we need to use the following command to install dependencies for PCB-AoI:
pip install examples/resources/algorithms/FPN_TensorFlow-0.1-py3-none-any.whl

However, a new problem has arisen: the package FPN_TensorFlow-0.1 requires dataclasses~=0.8.
This dependency is the original implementation of dataclass, which requires Python>=3.6, <3.7. Higher versions of Python have integrated dataclass into the standard library.
Therefore, we are unable to complete the installation of FPN_TensorFlow-0.1-py3-none-any.whl.
ERROR: Ignored the following versions that require a different python version: 0.7 Requires-Python >=3.6, <3.7; 0.8 Requires-Python >=3.6, <3.7; 10.0.0 Requires-Python >=3.8; 10.0.1 Requires-Python >=3.8; 10.1.0 Requires-Python >=3.8; 10.2.0 Requires-Python >=3.8; 10.3.0 Requires-Python >=3.8; 10.4.0 Requires-Python >=3.8; 3.6.0 Requires-Python >=3.8; 3.6.0rc1 Requires-Python >=3.8; 3.6.0rc2 Requires-Python >=3.8; 3.6.1 Requires-Python >=3.8; 3.6.2 Requires-Python >=3.8; 3.6.3 Requires-Python >=3.8; 3.7.0 Requires-Python >=3.8; 3.7.0rc1 Requires-Python >=3.8; 3.7.1 Requires-Python >=3.8; 3.7.2 Requires-Python >=3.8; 3.7.3 Requires-Python >=3.8; 3.7.4 Requires-Python >=3.8; 3.7.5 Requires-Python >=3.8; 3.8.0 Requires-Python >=3.9; 3.8.0rc1 Requires-Python >=3.9; 3.8.1 Requires-Python >=3.9; 3.8.2 Requires-Python >=3.9; 3.8.3 Requires-Python >=3.9; 3.8.4 Requires-Python >=3.9; 3.9.0 Requires-Python >=3.9; 3.9.0rc2 Requires-Python >=3.9; 3.9.1 Requires-Python >=3.9; 3.9.1.post1 Requires-Python >=3.9; 3.9.2 Requires-Python >=3.9
ERROR: Could not find a version that satisfies the requirement dataclasses~=0.8 (from fpn-tensorflow) (from versions: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6)
ERROR: No matching distribution found for dataclasses~=0.8

Python 3.8 and higher
Step 1. Ianvs Preparation
During the installation of sedna-0.4.1 using the following command, an issue with uvicorn~=0.14.0 was encountered.
$ pip install examples/resources/third_party/sedna-0.4.1-py3-none-any.whl

WARNING: Ignoring version 0.14.0 of uvicorn since it has invalid metadata:
Requested uvicorn~=0.14.0 from https://pypi.tuna.tsinghua.edu.cn/packages/bf/fe/a41994c92897b162c0c83e8ef10bec54ebdefbce3f3725b530d2091492ac/uvicorn-0.14.0-py3-none-any.whl (from sedna==0.4.1) has invalid metadata: .* suffix can only be used with `==` or `!=` operators
    click (>=7.*)
           ~~~~^
Please use pip<24.1 if you need to use this version

ERROR: Could not find a version that satisfies the requirement uvicorn~=0.14.0 (from sedna) (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.0.10, 0.0.11, 0.0.12, 0.0.13, 0.0.14, 0.0.15, 0.1.0, 0.1.1, 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.2.6, 0.2.7, 0.2.8, 0.2.9, 0.2.10, 0.2.11, 0.2.12, 0.2.13, 0.2.14, 0.2.15, 0.2.16, 0.2.17, 0.2.18, 0.2.19, 0.2.20, 0.2.21, 0.2.22, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.3.5, 0.3.6, 0.3.7, 0.3.8, 0.3.9, 0.3.10, 0.3.11, 0.3.12, 0.3.13, 0.3.14, 0.3.15, 0.3.16, 0.3.17, 0.3.18, 0.3.19, 0.3.20, 0.3.21, 0.3.22, 0.3.23, 0.3.24, 0.3.25, 0.3.26, 0.3.27, 0.3.28, 0.3.29, 0.3.30, 0.3.31, 0.3.32, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.4.5, 0.4.6, 0.5.0, 0.5.1, 0.5.2, 0.6.0, 0.6.1, 0.7.0b1, 0.7.0b2, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.8.6, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.10.4, 0.10.5, 0.10.6, 0.10.7, 0.10.8, 0.10.9, 0.11.0, 0.11.1, 0.11.2, 0.11.3, 0.11.4, 0.11.5, 0.11.6, 0.11.7, 0.11.8, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.13.4, 0.14.0, 0.15.0, 0.16.0, 0.17.0.post1, 0.17.1, 0.17.2, 0.17.3, 0.17.4, 0.17.5, 0.17.6, 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.20.0, 0.21.0, 0.21.1, 0.22.0, 0.23.0, 0.23.1, 0.23.2, 0.24.0, 0.24.0.post1, 0.25.0, 0.26.0, 0.27.0, 0.27.0.post1, 0.27.1, 0.28.0, 0.28.1, 0.29.0, 0.30.0, 0.30.1, 0.30.2, 0.30.3, 0.30.4, 0.30.5, 0.30.6)
ERROR: No matching distribution found for uvicorn~=0.14.0

After investigation, this is due to the removal of some old interfaces in the new version of pip.
A temporary solution is to clone the sedna repository locally and manually change the version of the uvicorn package in lib/requirements.txt to 0.15.0.
This issue has been reported to the Sedna project; for more details, please check https://github.com/kubeedge/sedna/issues/440.
Step 2. Dataset and Model Preparation
Apart from the issue with dataclasses~=0.8, FPN_TensorFlow-0.1-py3-none-any.whl also requires tensorflow~=1.14.0.
However, according to the version information on PyPI/Tensorflow, tensorflow==1.14.0 was released on June 19, 2019, and only supports versions of Python<=3.7. For versions above Python>3.7, it is necessary to use a version of tensorflow>=2.0.0.
If you attempt to install it in Python>=3.8 , the error message will be as follows:
$ pip install examples/resources/algorithms/FPN_TensorFlow-0.1-py3-none-any.whl

Processing ./examples/resources/algorithms/FPN_TensorFlow-0.1-py3-none-any.whl
Collecting wheel~=0.36.2 (from FPN-TensorFlow==0.1)
  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/65/63/39d04c74222770ed1589c0eaba06c05891801219272420b40311cd60c880/wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Collecting libs~=0.0.10 (from FPN-TensorFlow==0.1)
  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/60/65/01475e4fbf0d7539378019983c6d258e435ffbbfb497ceb6ec7fbea83eed/libs-0.0.10-py3-none-any.whl (5.8 kB)
INFO: pip is looking at multiple versions of fpn-tensorflow to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement tensorflow~=1.14.0 (from fpn-tensorflow) (from versions: 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0rc0, 2.6.0rc1, 2.6.0rc2, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.6.5, 2.7.0rc0, 2.7.0rc1, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.8.0rc0, 2.8.0rc1, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.8.4, 2.9.0rc0, 2.9.0rc1, 2.9.0rc2, 2.9.0, 2.9.1, 2.9.2, 2.9.3, 2.10.0rc0, 2.10.0rc1, 2.10.0rc2, 2.10.0rc3, 2.10.0, 2.10.1, 2.11.0rc0, 2.11.0rc1, 2.11.0rc2, 2.11.0, 2.11.1, 2.12.0rc0, 2.12.0rc1, 2.12.0, 2.12.1, 2.13.0rc0, 2.13.0rc1, 2.13.0rc2, 2.13.0, 2.13.1)
ERROR: No matching distribution found for tensorflow~=1.14.0

Summary
After testing several versions of Python, we found that none of them can run the existing Quick Start example.
The situation clearly demonstrates that outdated dependencies has already led to irreconcilable conflicts，indicating that resolving the dependency issues for Ianvs is very urgent and a sound mechanism for updating dependency versions needs to be established.

I conducted more experiments and found that things are a little tricky.
Dependencies conflicts between Ianvs Core and Examples/
Problems of Sedna
Ianvs Core needs sedna as a dependency. sedna-0.4.1-py3-none-any.whl describes its dependencies as follow：
numpy>=1.13.3  # BSD
colorlog~=4.7.2  # MIT
websockets~=9.1  # BSD
requests>=2.24.0  # Apache-2.0
PyYAML  # MIT
setuptools~=54.2.0
fastapi~=0.68.1  # MIT
pydantic>=1.8.1  # MIT
tenacity~=8.0.1  # Apache-2.0
joblib~=1.2.0  # BSD
pandas  # BSD
six~=1.15.0  # MIT
minio~=7.0.3  # Apache-2.0
uvicorn~=0.14.0  # BSD
pycocotools

The use of ~= locks some dependencies to older versions, while other dependencies in the user's example require newer versions. This creates significant conflicts.
For instance, sedna requires fastapi~=0.68.1, and pydantic is a dependency of fastapi. This restriction necessitates satisfying pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2.
However, some new examples (such as vllm in #122) require a dependency on pydantic >=2.0, which leads to severe conflicts and causes installation errors with pip.
Installing collected packages: pydantic
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
sedna 0.4.1 requires fastapi~=0.68.1, but you have fastapi 0.112.1 which is incompatible.

This is mainly due to two reasons:

Ianvs adopts the offline sedna-0.4.1-py3-none-any.whl installation wheel, which requires step-by-step environment configuration and prevents pip from resolving multiple package conflicts at once.
Sedna uses formats like ~= to describe dependency versions, causing some unimportant packages (such as uvicorn, fastapi, colorlog, websockets, etc.) to a fixed old version, leading to conflicts with the new packages introduced by users in the examples/, making it completely unusable for them.

Problems of Examples
Examples also have dependencies issues.
If you check PCB-AoI or other examples, you may find inside their README.md, there is usually an extra configuration guide for dependencies. This can be quite complex, involving the installation of offline wheels or self-modified packages.
What's more, different examples typically have varying dependencies, leading to complex dependency issues both between the example and Ianvs Core / Sedna, and among the examples themselves.
A Possible solution
Addressing these issues requires changes to the overall architecture of Ianvs, which may involve:

Separating core and examples into different repositories. The dependency differences among different examples are significant, and they target very specific scenarios, making it difficult for general users to benefit from these examples in getting started. I know that some examples may stem from related businesses, but if we want to expand the influence and usability of Ianvs as an open-source project, Ianvs might need to consider only retaining very general and user-friendly examples as a Quick Start.
Distributing ianvs and Sedna via PyPI, avoiding dependency issues caused by offline installation packages. This is a common practice among mainstream open-source projects and aligns better with user habits.

This plan may be somewhat radical. The final solution must be fully discussed within the community, which is beyond my current ability to resolve. If you're interested in addressing this issue, please feel free to comment.

Considering that I am working on the implementation of #96, I will temporarily put this issue on hold.

It's a good proposal, and I think we can discuss it at the regular community meeting.


Remove version: Jackett.Jackett version 0.21.2219

404 - release deleted
Microsoft Reviewers: Open in CodeFlow

URL: https://github.com/Jackett/Jackett/releases/download/v0.21.2219/Jackett.Installer.Windows.exe
Response Error: The remote server returned an error: (404) Not Found.
(Automated message - build 918)


Typo/Bug in InvokePlatformCodeDemos

I noticed a small bug in one of the demos.  I can push a branch for a fix.
Here is the issue (deviceOrientationService1 was copy-pasted instead of deviceOrientationService2 variable name, i think):
Link to repo source.
        var deviceOrientationService1 = new ConditionalCompilationDeviceOrientationService();
        DeviceOrientation orientation1 = deviceOrientationService1.GetOrientation();
        conditionalCompilationOrientationLabel.Text = orientation1.ToString();

        var deviceOrientationService2 = new PartialMethodsDeviceOrientationService();
        DeviceOrientation orientation2 = deviceOrientationService1.GetOrientation();
        partialMethodsOrientationLabel.Text = orientation2.ToString();

Closed by https://github.com/dotnet/maui-samples/pull/226


Make 3rd party libs being optional

Would be nice if the dependencies can be cut down to zero. features such as YAML file format, JSON or Metrics should go to optional modules.
For SLF4J there could be a factory that checks if slf4j is available and only then logging will be available.

My 2 cents
Agreed re: Metrics and I just came up with a possible PR I'll post later.
For YAML/JSON, I disagree - those are kind of cross cutting
For SLF4J I totally disagree - logging facades are totally acceptable to pull in the API for. Not the implementation, but the API is fine. SLF4J is used by virtually every major library created in the last 5 years or so

I took a quick look and there is no truly quick and dirty to  way (yes, you could use reflection or some sort of SPI) to remove the YAML - since it is used by the ClassPathSource. It seems more trouble than its worth since that encoding could be used by many providers


_headers ignored in getServerSideProps

Describe the bug
I have set headers from _headers file and it works well on the static assets but it doesn't work with pages created with getServerSideProps.
Here is the _headers file content:
/*
  # http security headers
  Strict-Transport-Security: max-age=63072000; includeSubDomains
  Cache-Control: max-age=31536000
  x-frame-options: SAMEORIGIN
  x-content-type-options: nosniff
  x-xss-protection: 1; mode=block
  referrer-policy: no-referrer-when-downgrade
  Access-Control-Allow-Origin: *

On Netlify Dashboard, I can see the message that "All header rules deployed without errors." and If I add HTTP Auth in this file then it starts working so the issue only belongs to the SSR pages.
I have also tried by configuring the headers from the netlify.toml and that also doesn't work. I have also tried to override the headers by using next.config.js which works well locally but doesn't with Netlify.
Versions

Next.js: 10.2.3

If you're using file-based installation

netlify.toml:[build]
  functions = "functions"
  # Default build command.
  command = "npm run build"

[[plugins]]
  package = "@netlify/plugin-nextjs"

[[plugins]]
  package = "netlify-plugin-cache-nextjs"

hey @samiahmedsiddiqui! i believe this is related to https://github.com/netlify/netlify-plugin-nextjs/issues/156. a temporary workaround would be for you to patch-package the plugin to remove https://github.com/netlify/netlify-plugin-nextjs/blob/main/src/lib/templates/functionBase.js#L27. as for a permanent solution, we'd need to reopen the conversation about removing this line or fixing upstream in the framework. i'll keep you updated!

@lindsaylevine Thank you for your response. Please also notice that I am talking about all the headers not only the no-cache header as discussed here:
https://github.com/netlify/netlify-plugin-nextjs/issues/156
Please keep me updated about the progress, Thank you!


fix serialization in ContextElementDependency

What kind of change does this PR introduce?
fixes #15177

Did you add tests for your changes?
not yet

Does this PR introduce a breaking change?
no

What needs to be documented once your changes are merged?
nothing

Thanks


Add utils to detect .NET project

Similar to the utils to detect if the project is a java project, there should be a similar utils to detect if the project is a .NET/C# project.  This will help other extensions flow their logic based on this information better.

@timheuer are you talking about these java utils? https://github.com/Microsoft/vscode-azuretools/blob/master/appservice/src/utils/javaUtils.ts#L55 As @nturinski mentioned here, we actually don't want those utils to live in this package. How are you planning on using it? Are you working on another extension?

yeah something similar.  There are other Azure extensions that would need to know some more context in order to kick off different settings.  Having an 'am i a .NET project' capability in a core location will help other Azure extensions and not reproduce that multiple places.
Unfortunately the workspaceContains/onLanguage triggers aren't enough because those are launch/activation triggers and not something that can be queries later.

Can you give examples of extensions? The current java utils are in an app service package, and so only app service extensions should be using it. I don't think java utils or c# utils are specific to app service, which is why we only reluctantly added them in the first place (and why we plan to remove them).
I agree a common capability could be useful, but I want to avoid these packages becoming a dumping grounds for random utils.

If the plan is to remove the javaUtils, that makes sense.  This would be for other app service usage as well.  We just had a discussion and think we'll own our own dotnetUtils extension...closing.


Use master for build deps for docs

Summary
There is an issue where some of the old tags cannot be build using the dependencies in those tags.  Instead build all tags from the dependencies defined in master. We can revisit if this breaks in the future.
Impact
Webpage should be able to be built again from CI.
Testing
This PR passing.

@hartmannathan  or @xiaoxiang781216  perhaps you can review this to get the webpage building again.

Merged as part of #77


Added resetLevel method

This provides a resetLevel() method as an alternative solution to the use cases detailed in #172

Yep, I think this looks great, merged. Thanks for the contribution! I'll put out a v1.8.0 release with this included in just a moment.


Run directories

Workflows in production are often run in the same directory multiple times. Various log files, checkpoint files, submit scripts, config options etc are often produced and ideally stored in a manner that would allow the user to easily identify files associated with a single workflow run. Having a run specific directory would also help #39 and #48.
I propose that we have run directory that is named runNNN by default and placed in the current directory. Users would have the option to specify a different run directory path, if they so choose.

The idea seems good, but some details are not clear to me:

What is NNN?
Is it always three characters?

I should have specified this as run[0-9][0-9][0-9] (Run followed by three digits).
In swift we had 3 digits. We need to guess here in terms of how many instances we expect users to run.

Specifically the logic here would be that the first run would have the dir run000 and every subsequent run would increment the run number by 1. The only reason to use runNNN is that it looks nicer with the number padded with 0's when ls'ing the directories.

what happens if there are more than 1000 runs?

run000 -> run001 - > ... run999 -> run1000.
It's padded to 3 positions.

sounds reasonable

From Slack, @danielskatz suggested a top directory with the numbered ones underneath, and now we're all in agreement on:
runinfo/000
runinfo/001
...

Great! I'll add this to the multisite branch that needs this functionality.

This functionality has now been merged to master from branch:multisite.


Clean various lint warnings

Fix pylint no else return.
Fix pylint unused argument.
Fix bad variable naming, eg incorrect casing.
Fix pylint redefined outer name.
FIx pylint len as condition.
Disable some warnings when we want to keep it as is, eg # pylint: disable=redefined-builtin.

This PR is preferably reviewed with white-space changes hidden, since many lines are affected by removing else: statement, although it's mostly a white-space change.

looks good.


FinRL_Ensemble_StockTrading_ICAIF_2020.ipynb vs ICAIF 2020 paper

Hi,
thank you very much for sharing this great project. Veyr impressive work!
I have a question: It seems the portfolio results in the jupyter notebook
https://github.com/AI4Finance-Foundation/FinRL/blob/master/FinRL_Ensemble_StockTrading_ICAIF_2020.ipynb
are quite a bit worse than the results shown on
https://towardsdatascience.com/deep-reinforcement-learning-for-automated-stock-trading-f1dad0126a02
as well as in the ICAIF 2020 paper (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3690996).
Why is that? Is that a different version and if so in which respect?
Thank you very much!!

The answer to your question can have numerous answers.
A few examples:

Hyperparameter optimization method
Which data was trained upon
Which data was tested upon
Which out-of-sample data was used
Model settings
....
Market conditions

The code given in the notebook is not the identical version of the code corresponding to the paper. The notebooks are just a part of a complete application.

This notebook seems to have been removed:
https://github.com/AI4Finance-Foundation/FinRL/blob/master/FinRL_Ensemble_StockTrading_ICAIF_2020.ipynb
Is this available somewhere else?

Please check this folder: https://github.com/AI4Finance-Foundation/FinRL/tree/master/tutorials
Thanks.

It appears that this link has also been deprecated:
https://github.com/AI4Finance-Foundation/FinRL/tree/master/tutorials
Is there a new place i can find it?


Text Highlighting for Changes YAML

In this lab, there are several places where we are asked to change our current YAML to the YAML shown in the lab. The specific differences are summarized, but not specifically called out.
I can simply copy the YAML from the lab notes and use it to replace the entire YAML file that already exists locally on my desktop, but then I wouldn't really learn how the changes impact the application's behavior.
It would be easier (and more educational) if the entire YAML were shown (as is currently done), but the needed changes were shown in a different color.

Currently, we are limited to what GitHub's Markdown engine can support. There is no format that supports both syntax and arbitrary line highlighting available at this time. There is diff highlighting, however, we would lose the YAML syntax highlighting, and there would be + and - symbols at the beginning of the line that would need to be edited out by the student manually.

we cannot achieve the change highlighting with the current rendering options.  We will look into this for a future lab framework.


Short backtrace

I have large codebase which uses anyhow::{Result, Context, bail} a lot,
and the errors don't tell much. No filename/line numbers at all.
Using RUST_BACKTRACE=1 is not convenient.
I found https://github.com/dtolnay/anyhow/issues/22 but it will require many changes to add it everywhere.
Is there an elegant way to get short backtrace when error occur, just like we get it with unwrap()?
Example:
use anyhow::{Result, bail};

fn main() -> Result<()>  {
    job()
}

fn job() -> Result<()> {
    bail!("error in job")
}

cargo run
Error: error in job

Imagine large codebase with many errors coming from internal libraries... no idea where it happens.
Thanks

You can call .backtrace() on the error value to get a backtrace you can print if RUST_BACKTRACE=1 or RUST_LIB_BACKTRACE=1 is used.

it used Rust std::backtrace, so there is no direct access to frames, only std::fmt::Debug which is verbose


Overhaul: ES6, ESLint, package.json, some smaller fixes

Supersedes #11.
I used a simpler UMD-style function without AMD, because browserify/webpack don’t need that, and ESLint only works with a IIFE.
all commits are rather small, except for the ESLint style thing, so just look at them individually, please.
I also fixed a bug that occurs if e.g. $.fn.dataTable doesn’t exist (only DataTable with “D”)

Whoaaa!
I'm planed to move to ES6 in month, but you 'very already done it. Great job!
I'll look at next week


revert cma when uninstall globalhub

Summary
Related issue(s)
Fixes #
Tests

[ ] Unit/function tests have been added and incorporated into make unit-tests.
[ ] Integration tests have been added and incorporated into make integration-test.
[ ] E2E tests have been added and incorporated into make e2e-test-all.
[ ] List other manual tests you have done.

/hold

/retest

/unhold


can't marry another wife

I use marry anyone mod with bannerkings and with the marriage overhaul I can't choose myself as a spouse in the proposal.
I have enabled polygamy in mcm

The Bannerkings marriage system is incompatible with multiple marriages. I have used suemorespouses that has a separate marriage mechanic that has worked for me, but no promises.


pull request for e2e test at 01/14/2017 06:15:13

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Ensure X509KeyManager methods are called on the correct time when using server-side and support more methods of ExtendedSSLSession.

Motivation:

Before when on server-side we just called the X509KeyManager methods when handshake() was called the first time which is not quite correct as we may not have received the full SSL hello / handshake and so could not extra for example the SNI hostname that was requested.
OpenSSL exposes the SSL_CTX_set_cert_cb function which allows to set a callback which is executed at the correct moment, so we should use it. This also allows us to support more methods of ExtendedSSLSession easily.

Modifications:

- Make use of new methods exposed by netty-tcnative since https://github.com/netty/netty-tcnative/pull/388 to ensure we select the key material at the correct time.
- Implement more methods of ExtendedOpenSslSession
- Add unit tests to ensure we are able to retrieve various things on server-side in the X509KeyManager and so verify it is called at the correct time.
- Simplify code by using new netty-tcnative methods.

Result:

More correct implementation for server-side usage and more complete implemented of ExtendedSSLSession.

This depends on https://github.com/netty/netty-tcnative/pull/388 and https://github.com/netty/netty/pull/8281

@carl-mastrangelo PTAL again

I will merge this once the CI is done


remove unused social widget content script

Reviewer: @jdorweiler
cc @russellholt
Description:
Remove calls to unused social-widgets content script.
Steps to test this PR:

after reloading extension in chrome, visit any website and open devtools. Navigate to the 'sources' tab, and on the left navigate to the 'content scripts' tab. There shouldn't be anything there.
To really verify, try testing the code snippet feature on stackoverflow which was breaking due to frame access errors stemming from this content script. Click the 'run code snippet' button in the second answer here: https://stackoverflow.com/questions/50119240/why-this-recursive-javascript-function-returns-undefined

👍


Manually added attacks break things

Suspect player used "Add attack".   This is a legacy feature that we should probably remove "Add attack", "Add defense", Add Skill", etc  at some point.  The item.system.XMLID is undefined and so are all the other HD XML props, which are basically required with newer code base.  Specifically the migration throws lots of warnings.  Also some when actor sheets are opened.
Consider adding XMLID = "CUSTOMATTACK" for attack items missing XMLID.  This should at least provide basic support for migrations and opening actor/item sheets.

The STR items can be reproduced with STRIKE maneuver and modifying effective STR.
The KNOCKBACK needs to be supported in the attack/damage chain.  Even this hack as an attack uses END, which isn't correct.  A checkbox on the attack to not use END would be pretty hacky.  Could also be reproduced with a STRIKE maneuver.  Create a HD power (HANDATTACK) for KNOCKBACK that costs no END, then import it.

Long term provide features that remove the need for these custom attacks.
Some issues are related to super old actors, we don't even have the system version number.  Suggest we skip migration on those actors, perhaps a single warning about re-uploading that actor.  Don't spam the GM during migrations.  Raymond in particular never deletes anything and migrations take several minutes as is.  The warnings make it take even longer.

{
    "name": "STR",
    "type": "attack",
    "system": {
        "description": "",
        "effect": "",
        "cost": 0,
        "modifiers": [],
        "hdcid": "",
        "parentid": "",
        "defense": "pd",
        "killing": false,
        "piercing": 0,
        "penetrating": 0,
        "advantages": 0,
        "dice": 43,
        "toHitMod": 0,
        "extraDice": "zero",
        "uses": "ocv",
        "targets": "dcv",
        "end": 0,
        "stunX": 0,
        "class": "physical",
        "areaOfEffect": {
            "type": "none",
            "value": 0,
            "width": 0,
            "height": 0
        },
        "stunBodyDamage": "stunbody",
        "ocv": null,
        "dcv": null,
        "knockbackMultiplier": "0",
        "usesStrength": false,
        "usesTk": false,
        "noHitLocations": false,
        "basePointsPlusAdders": null,
        "activePoints": 0,
        "realCost": "0",
        "charges": {
            "value": null,
            "max": null,
            "recoverable": false
        },
        "USE_END_RESERVE": false,
        "max": 0,
        "value": 0,
        "baseCost": 0,
        "activePointsDc": 0
    },
    "_id": "KxYsSS8d8dPZdCB3",
    "img": "icons/svg/sword.svg",
    "effects": [],
    "folder": null,
    "sort": 0,
    "ownership": {
        "default": 0,
        "vyaLXzFbhxhlGFym": 3
    },
    "flags": {},
    "_stats": {
        "systemId": "hero6efoundryvttv2",
        "systemVersion": "3.0.63",
        "coreVersion": "11.315",
        "createdTime":<PHONE_NUMBER>051,
        "modifiedTime":<PHONE_NUMBER>101,
        "lastModifiedBy": "wjwBVu3E3fGldQSo"
    }
}

Closing. 3.0.64 provides warning messages to users and CHANGELOG.md entry.


Syntax highlighting not working correctly

Summary
As a side effect of #46, the current documentation syntax highlighting is not working as expected some parts get syntax highlighting but all that include let are not, I've attached the screenshot below for reference.
Preview

Thanks @IvanBila it looks like our syntax highlighter is not ready for modern JavaScript, I will follow up with the documentation team and see if we can update it.  Hopefully it's not a hard change to make.

Hopefully it's not a hard change to make.

Well turns out it's an extremely hard change to make because of silly internal Google things.  But I am working on it.

@samtstern Is it possible to maybe revert the commit that uses let until this issue is fixed? let me know what is your opinion.

@IvanBila I think I'd prefer the nicer code over the nicer colors

@samtstern I also agree on that, talking about nice code, (OFF TOPIC)I wanted to add async/await as an extra part of the docs(maybe at the bottom), where I just show how to do the same thing using async await. What is your opinion on this?

@IvanBila looks like all instances of let have been replaced with const and async/await has been added in #121
Closing this now


[BUG] Even with flow control enabled, carbon aggregator drops incoming metrics when output queues are full

There is a config setting, USE_FLOW_CONTROL, which defaults to enabled. The stated behaviour is that, when enabled, carbon will pause reading metrics when its sink is full. Indeed, carbon does pause receiving metrics, but it also drops any "in-flight" metrics.
Test setup
Set up Carbon to not relay aggregator misses, and to use line protocol (so it's easy to test from bash). Also set up a.* to aggregate every 3 seconds, and carbon to spit out stats every 1 second, just to avoid waiting around. And verify that FLOW_CONTROL is True, so that data shouldn't be dropped when buffers fill.
~/carbon$ (cd test-root/conf; for i in *.conf; do diff -u0 ../../conf/$i.example $i; done)
--- ../../conf/aggregation-rules.conf.example   2021-10-08 18:21:23.195248379 +0100
+++ aggregation-rules.conf      2024-08-21 13:31:47.789993319 +0100
@@ -59,0 +60,2 @@
+
+a.<a> (3) = sum a.<a>
--- ../../conf/carbon.conf.example      2024-07-08 17:43:30.486176214 +0100
+++ carbon.conf 2024-08-21 14:35:32.558819082 +0100
@@ -436 +436 @@
-DESTINATIONS = <IP_ADDRESS>:2004
+DESTINATIONS = <IP_ADDRESS>:2003
@@ -441 +441 @@
-# DESTINATION_PROTOCOL = pickle
+DESTINATION_PROTOCOL = line
@@ -614 +614,2 @@
-DESTINATIONS = <IP_ADDRESS>:2004
+DESTINATIONS = <IP_ADDRESS>:2003
+DESTINATION_PROTOCOL = line
@@ -668 +669 @@
-# CARBON_METRIC_INTERVAL = 60
+CARBON_METRIC_INTERVAL = 1
@@ -680,0 +682 @@
+LOG_AGGREGATOR_MISSES = False
~/carbon$ grep ^USE_FLOW_CONTROL test-root/conf/carbon.conf
USE_FLOW_CONTROL = True
USE_FLOW_CONTROL = True
USE_FLOW_CONTROL = True

Fire up a socket to look at what comes out:
~/carbon$ nc -l 2003 | grep -i drop

Fire up an aggregator:
~/carbon$ GRAPHITE_ROOT=$HOME/carbon/test-root/ PYTHONPATH=lib test-env/bin/twistd --nodaemon --reactor=epoll --no_save carbon-aggregator

Send 30k unaggregated metrics:
~/carbon$ date=1724239620; for i in {100001..130000}; do echo b.$i 1000 $date; done > test-data; nc -q0 localhost 2023 < test-data
1724239620

This fairly consistently gives me stats like:
carbon.aggregator.hostname-a.destinations.127_0_0_1:2003:None.fullQueueDrops 14500<PHONE_NUMBER>

(If I make a longer metric name, the count drops, so this looks related to a buffer size at some point in the pipeline.)
Send 80k aggregated metrics:
~/carbon$ date=1724239620; for i in {100001..180000}; do echo a.$i 1000 $date; done > test-data; nc -q0 localhost 2023 < test-data

This gives much less consistent output, but does consistently drop data anyway. The timestamps are later than in the unaggregated case, so the dropping is at aggregation time, not production time:
carbon.aggregator.hostname-a.destinations.127_0_0_1:2003:None.fullQueueDrops 4369<PHONE_NUMBER>
carbon.aggregator.hostname-a.destinations.127_0_0_1:2003:None.fullQueueDrops 1503<PHONE_NUMBER>
carbon.aggregator.hostname-a.destinations.127_0_0_1:2003:None.fullQueueDrops<PHONE_NUMBER>640

Send 1 aggregated metric with 80k datapoints:
~/carbon$ date=1724239620; for i in {100001..180000}; do echo a.100000 1000 $date; done > test-data; nc -q0 localhost 2023 < test-data

This never drops anything, even with very, very large counts.
There are a few problem cases:

Where the aggregator passes through metrics verbatim (under FORWARD_ALL), acting as a relay, when the MAX_QUEUE_SIZE is hit, the cacheFull event is called. However, this apparently only pauses reading from the socket, but doesn't pause processing of already-read data. This means it turns on flow control, but drops a bunch of metrics on the floor immediately. This bug probably affects the relay service, too.
Where the aggregator acts as an aggregator, it will fill its buffers with as many IntervalBuffer pairs as it can, then aggregate out the intervals when their LoopingCalls tick over. The aggregation process is not paused at all when the buffer fills, which means, again, metrics are just dropped on the floor when the output becomes full. Realistically, the problem here is that the aggregator's state isn't properly limiting its count of IntervalBuffer pairs to MAX_QUEUE_SIZE -- if it ever exceeds this value, it is likely to drop data on the next aggregation tick, so should be considered independently full.
Curiously, there is no bound on the amount of data inside a given buffer, nor on the number of buffers, so in the case where the test generates tens of thousands of datapoints which all aggregate into one value, the aggregator dutifully generates a single list with all of the datapoints in RAM and then spits them out in one go, later. There's no failure in my test, then, but the size of the aggregator is in theory unbounded in this case!

Inspecting the code, the same logic exists in _MetricCache.store -- it relies on events.cacheFull() to prevent any more calls, and will drop any datapoints after this until the queue drains somewhat.
Expected behavior
With USE_FLOW_CONTROL = True, no metrics should ever be dropped.
Suggestion
I propose that:

USE_FLOW_CONTROL should be properly implemented for daemons other than the cache. Fixing this reduces the number of drops in my example from 14500 to 6332 but not to zero. It's also very simple to do.
MAX_QUEUE_SIZE should be (allowed to be?) taken as a soft limit which causes the input streams to pause, and the daemon be allowed to exceed this limit perhaps by some factor once the inputs are paused -- or it should soft-cap below MAX_QUEUE_SIZE. Perhaps QUEUE_LOW_WATERMARK_PCT should be lowered, and QUEUE_HIGH_WATERMARK_PCT should be added, with the queue considered full above this but not actually dropping data. Experimentation suggests QUEUE_LOW_WATERMARK_PCT = 0.6 and QUEUE_HIGH_WATERMARK_PCT = 0.7 are sufficient to avoid metric dropping on the default MAX_QUEUE_SIZE = 10000. Setting 0.6/0.8 was not sufficient. Alternatively, HARD_MAX_QUEUE_SIZE_PCT = 1.25 (taking MAX_QUEUE_SIZE as the high watermark) and the default QUEUE_LOW_WATERMARK_PCT = 0.8 was fine, too.
The same, or similar, massaging should be applied to MAX_CACHE_SIZE.

These will fix the relay behaviour.

The aggregator's IntervalBuffer count should be capped to some function of MAX_QUEUE_SIZE or some new setting, and behave like a full queue when it exceeds this. Flow control might not be the correct solution here; the aggregation buffer count can never be "caught up" but is instead a function of the number of active metrics.
There should be some limit on the number of datapoints per IntervalBuffer and/or the total number of datapoints shared between all of these.

The fix for the aggregation case is difficult, as the code currently only understands when its sole sink (the relay output, in this case) is blocked. The logic needs to understand that it can be paused for multiple reasons. These might best be moved to a distinct ticket.

Wow, that's really impressive investigation and fix, much appreciated!
Let me check and merge patches.


[PIO-59] Use /dev/urandom to create access keys.

Creating access keys based on /dev/random may take a long time
on virtualized or clound environment for example more than
10 minutes.

I am convinced that this PR does not decrease the security of generating random app access keys. I will merge if there is no objection.


auth: store cached permission information in a form of interval tree

This commit change the type of cached permission information from the
home made thing to interval tree. It improves computational complexity
of permission checking from O(n) to O(lg n).
/cc @heyitsanthony This is a preparation for https://github.com/coreos/etcd/issues/7468 , could you take a look? We can support the unlimited range by configuring a permission like ["a", "\xff") in a simple manner thanks to interval tree.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@ff6d686). Click here to learn what that means.
The diff coverage is 55.76%.

@@            Coverage Diff            @@
##             master    #7569   +/-   ##
=========================================
  Coverage          ?   70.15%
=========================================
  Files             ?      320
  Lines             ?    26165
  Branches          ?        0
=========================================
  Hits              ?    18355
  Misses            ?     6355
  Partials          ?     1455

Impacted Files
Coverage Δ

auth/store.go
83.46% <100%> (ø)

auth/range_perm_cache.go
52.5% <54.9%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ff6d686...71da099. Read the comment docs.

@heyitsanthony updated based on your comments. The main changed points:

now the code doesn't use StringComparable instead of StringAffineInterval because "" shouldn't be larger than other strings
changed if maxEnd.Compare(n.ivl.End) > 0 { maxEnd = n.ivl.End } in the callback to n.Ivl.End.Compare(maxEnd) > 0

But this PR depends on your PR https://github.com/coreos/etcd/pull/7581 so it cannot be merged now.

@heyitsanthony rebased on the master (that includes your changes of interval tree), so this PR would be able to work well. Could you take a look?

@heyitsanthony two test fails can be found but they don't seem to be related with this change because they are related to membership management: https://github.com/coreos/etcd/issues/7594 and https://github.com/coreos/etcd/issues/7595

@heyitsanthony kindly ping? Do you have any opinions about the failed tests?

@mitake oops sorry for losing this in the shuffle. I kicked CI and it's passing.
The interval tree adt could stand to have contiguity checking and BytesAffineInterval to save allocs from the string casts, but that's not a blocker.
lgtm, thanks!

@heyitsanthony then I'm merging it, thanks! I also add the changes of interval tree in other PRs.
BTW the failed test seems to be same with https://github.com/coreos/etcd/issues/7595


Error in using spaghetti.element_as_gdf

Describe the bug
Error while using spaghetti.element_as_gdf
To Reproduce
Run the following code that was tasked from the library tutorial:
%config InlineBackend.figure_format = "retina"
%load_ext watermark
%watermark

import esda
import libpysal
import matplotlib
import matplotlib_scalebar
from matplotlib_scalebar.scalebar import ScaleBar
import numpy
import spaghetti
import splot

%matplotlib inline
%watermark -w
%watermark -iv

ntw = spaghetti.Network(in_data=libpysal.examples.get_path("streets.shp"))
ntw

_, arc_df = spaghetti.element_as_gdf(ntw, vertices=True, arcs=True)
arc_df.head()

System & Versions

Platform information:
posix darwin
posix.uname_result(sysname='Darwin', nodename='eduroam-int-dhcp-97-26-155.ucl.ac.uk', release='22.2.0', version='Darwin Kernel Version 22.2.0: Fri Nov 11 02:04:44 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T8103', machine='x86_64')

Python version:
3.10.9 (main, Mar  1 2023, 12:33:47) [Clang 14.0.6 ]

spaghetti version:
1.6.8

libpysal version:
4.6.2

SciPy version:
1.10.0

NumPy version:
1.23.5

GeoPandas version (if applicable):
0.13.0

More info:
I received the following error in the this line of code:
_, arc_df = spaghetti.element_as_gdf(ntw, vertices=True, arcs=True)

AttributeError: You are calling a geospatial method on the GeoDataFrame, but the active geometry column to use has not been set.
There are no existing columns with geometry data type. You can add a geometry column as the active geometry column with df.set_geometry.

Thanks for the report!
This error has been fixed in the latest release of spaghetti. Please upgrade to spaghetti==1.7.4.
xref: https://github.com/pysal/spaghetti/issues/717, https://github.com/pysal/spaghetti/issues/718

@jGaboardi yeah!! thank you soo much!!!


GEODE-9492: Ignore default port test in stress tests

PROBLEM
GeodeRedisServerStartupDUnitTest.startupOnDefaultPort() tests the
system's use of a fixed port. Repeat test tasks run multiple instances
of this test concurrently. Most of the concurrently running instances of
the test are likely to fail when they find that port already in use.
SOLUTION

Add an IgnoreInRepeatTestTasks test category.
Configure repeat tests to exclude tests with the
IgnoreInRepeatTestTasks category.
Add the IgnoreInRepeatTestTasks category to
GeodeRedisServerStartupDUnitTest.startupOnDefaultPort()

That seems reasonable. Odd that this is the first we have seen of this kind of problem. One would have thought other default ports would have this problem. But perhaps, this test is flawed. Perhaps breaking it into a test of the default port number and a test of the starting up on a specified port would be more appropriate then altering our test framework?

@mhansonp I think it started appearing because Dale removed use of Docker for these tests.

Odd that this is the first we have seen of this kind of problem. One would have thought other default ports would have this problem.

It's a consequence of two things. First, we are no longer dockerizing test workers. Until a few weeks ago, each test worker ran in a separate docker container, and within each container fewer processes were competing for ports. Now that dozens of concurrently running tests compete for the same ports, we will see more port conflicts like this.
Second, very few tests use default ports, so there are likely to be few conflicts… except in stress tests, which pit multiple instances of a test against each other.

But perhaps, this test is flawed. Perhaps breaking it into a test of the default port number and a test of the starting up on a specified port would be more appropriate then altering our test framework?

I chose this approach because it affects only tests that explicitly test the use of default ports. And my guess (or perhaps merely my hope) is that very few tests do that: Only one per service. So very few tests will use this technique to avoid stress. I hope.
As for altering the test framework, my assessment (for now) is that including these few tests in stress test runs isn't worth the extra complexity in the test framework.

As for altering the test framework, my assessment (for now) is that including these few tests in stress test runs isn't worth the extra complexity in the test framework.

I suggested that we modify this test to be repeated test capable rather than adding a new category which you have done.
E.g. Pulling the default port in into a default port test. Then modify this code to test that it works fine with a configured port. I am not very attached to this idea, but it seems cleaner. Because then the redis code will get repeat testing.

I suggested that we modify this test to be repeated test capable rather than adding a new category which you have done.

Ah. I misread "then" as meaning "and then," and my misreading changed the entire meaning of what you wrote. Sorry about that.