HDDS-1629. Tar file creation can be optional for non-dist builds

Ozone tar file creation is a very time consuming step. I propose to make it optional and create the tar file only if the dist profile is enabled (-Pdist)
The tar file is not required to test ozone as the same content is available from hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT which is enough to run docker-compose pseudo clusters, smoketests.
If it's required, the tar file creation can be requested by the dist profile.
On my machine (ssd based) it can cause 5-10% time improvements as the tar size is ~500MB and it requires a lot of IO.
See: https://issues.apache.org/jira/browse/HDDS-1629

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
46
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
593
trunk passed

+1
compile
312
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
1766
branch has no errors when building and testing our client artifacts.

+1
javadoc
174
trunk passed

_ Patch Compile Tests _

+1
mvninstall
506
the patch passed

+1
compile
293
the patch passed

+1
javac
293
the patch passed

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
1
The patch has no ill-formed XML file.

+1
shadedclient
710
patch has no errors when building and testing our client artifacts.

+1
javadoc
173
the patch passed

_ Other Tests _

-1
unit
267
hadoop-hdds in the patch failed.

-1
unit
2088
hadoop-ozone in the patch failed.

+1
asflicense
50
The patch does not generate ASF License warnings.

6277

Reason
Tests

Failed junit tests
hadoop.hdds.scm.block.TestBlockManager

hadoop.ozone.TestMiniChaosOzoneCluster

hadoop.ozone.client.rpc.TestFailureHandlingByClient

hadoop.ozone.client.rpc.TestBCSID

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-887/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/887

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml

uname
Linux 2c094619b49a 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 4530f45

Default Java
1.8.0_212

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-887/1/artifact/out/patch-unit-hadoop-hdds.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-887/1/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-887/1/testReport/

Max. process+thread count
5338 (vs. ulimit of 5500)

modules
C: hadoop-ozone/dist U: hadoop-ozone/dist

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-887/1/console

versions
git=2.7.4 maven=3.3.9

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.


[v2.9] Fix post clean up steps

Issue:

Problem
  Updating the rancher version to have the new shepherd, but there was an issue with cleaning the rancher instance if there has been error.
Solution
 If there is an issue with cleaning up clusters, delete the corral regardless so there aren't any leftover resources.
Testing

Engineering Testing
Manual Testing

Automated Testing

Test types added/modified:

Unit
Integration (Go Framework)
Integration (v2prov Framework)
Validation (Go Framework)
Other - Explain: EXPLAIN
None
REMOVE NOT APPLICABLE BULLET POINTS ABOVE

If "None" - Reason: EXPLAIN THE REASON

If "None" - GH Issue/PR: LINK TO GH ISSUE/PR TO ADD TESTS

Summary: TODO
QA Testing Considerations

Regressions Considerations

TODO
Existing / newly added automated tests that provide evidence there are no regressions:

TODO

Backports:

2.8 -> https://github.com/rancher/rancher/pull/44596
2.7 -> https://github.com/rancher/rancher/pull/44595


Running backend produces garbage due to using colors 1.4.2

The colors 1.4.2 package produces garbage output and causes an infinite loop as shown here.
Expected Behavior
Doing a yarn start on the backend would start the backend correctly.
Current Behavior
Doing a yarn start on the backend causes garbage output and the terminal session to eventually lock up.
Possible Solution
Lock the version of the colors library to 1.4.0, which is unaffected.  Currently the backend is using 1.4.2.

yarn why v1.22.11
[1/4] 🤔  Why do we have the module "colors"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
=> Found<EMAIL_ADDRESS>info Reasons this module exists

"project#app#cypress#cli-table3" depends on it
Hoisted from "project#app#cypress#cli-table3#colors"
info Disk size without dependencies: "116KB"
info Disk size with unique dependencies: "116KB"
info Disk size with transitive dependencies: "116KB"
info Number of shared dependencies: 0
=> Found<EMAIL_ADDRESS>info This module exists because "project#backend#@backstage#backend-common#logform" depends on it.
info Disk size without dependencies: "120KB"
info Disk size with unique dependencies: "120KB"
info Disk size with transitive dependencies: "120KB"
info Number of shared dependencies: 0
✨  Done in 1.01s.

Steps to Reproduce

Clone the current release
Try and do a yarn start in packages/backend

Your Environment

OS:   Darwin 21.2.0 - darwin/x64
node: v16.13.0
yarn: 1.22.11
cli:  0.10.5 (installed)
Dependencies:
@backstage/app-defaults                                  0.1.3
@backstage/backend-common                                0.10.2
@backstage/backend-tasks                                 0.1.2
@backstage/catalog-client                                0.5.3
@backstage/catalog-model                                 0.9.8
@backstage/cli-common                                    0.1.6
@backstage/cli                                           0.10.5
@backstage/config-loader                                 0.9.1
@backstage/config                                        0.1.11
@backstage/core-app-api                                  0.3.1
@backstage/core-components                               0.8.3
@backstage/core-plugin-api                               0.4.1
@backstage/errors                                        0.1.5
@backstage/integration-react                             0.1.17
@backstage/integration                                   0.7.0
@backstage/plugin-api-docs                               0.6.21
@backstage/plugin-app-backend                            0.3.21
@backstage/plugin-auth-backend                           0.6.1
@backstage/plugin-catalog-backend                        0.19.4
@backstage/plugin-catalog-import                         0.7.8
@backstage/plugin-catalog-react                          0.6.10
@backstage/plugin-catalog                                0.7.7
@backstage/plugin-github-actions                         0.4.30
@backstage/plugin-org                                    0.3.33
@backstage/plugin-permission-common                      0.3.0
@backstage/plugin-permission-react                       0.2.1
@backstage/plugin-proxy-backend                          0.2.15
@backstage/plugin-scaffolder-backend-module-cookiecutter 0.1.7
@backstage/plugin-scaffolder-backend                     0.15.19
@backstage/plugin-scaffolder-common                      0.1.2
@backstage/plugin-scaffolder                             0.11.17
@backstage/plugin-search-backend-node                    0.4.3
@backstage/plugin-search-backend                         0.3.0
@backstage/plugin-search                                 0.5.4
@backstage/plugin-tech-radar                             0.5.1
@backstage/plugin-techdocs-backend                       0.12.2
@backstage/plugin-techdocs                               0.12.13
@backstage/plugin-user-settings                          0.3.15
@backstage/search-common                                 0.2.1
@backstage/techdocs-common                               0.11.2
@backstage/test-utils                                    0.2.1
@backstage/theme                                         0.2.14
@backstage/types                                         0.1.1
@backstage/version-bridge                                0.1.1
✨  Done in 0.76s.

Aw damn. So we did get bitten by this. Alright, of course let's try to get that fixed. It's very late now but we'll get on it.

Ok, @ctudball if this was a new npx @backstage/create-app, then I think if you delete the yarn.lock and run yarn again it should fix it as the dependency is locked in logform now. We will do a release on thursday to force the latest version of logform in backend-common which locks it too.
https://github.com/winstonjs/logform/pull/128 should have fixed it in new create-app installs.
We're waiting for this one too: https://github.com/cypress-io/cypress/pull/19622
You could add a resolutions block in your root package.json to force colors to 1.4.0 but hopefully might not need that as it should be locked for most people already.

https://github.com/backstage/backstage/pull/8836 is the pinned logform release which will go out on Thursday.


Fix checking of %%MatrixMarket matrix coordinate pattern symmetric

The %%MatrixMarket matrix coordinate pattern symmetric graphs don't have a weight column, which is accounted for in the helpers.h file.  However, a sanity test is conducted to evaluate whether the number of values parsed is equal to the number of edges times 3.  In the pattern graph, this produces an error since the number of "elements_per_line" for each edge is 2, not 3.  Therefore, I moved up the initialization of the "elements_per_line" variable and use this variable as opposed to a hardcoded 3 when checking sanity of the parser.

If you would like to add an example file can you just write up a small one that does not require not require citation for ease of use if anybody wants to use or download the code with different licenses?

I handwrote a graph of three stars, connected by one edge each.

Thanks so much for the fixes


"failed to execute default web browser"

I'm using Ubuntu 17.10, and the debian package from ppa:alessandro-strada/ppa
Following the installation instructions, I'm stopped at the step "The first time, you can run google-drive-ocamlfuse without parameters."  I get a dialog box pop up that says "Failed to execute deafult Web Browser \ Input/output error."
I ran it again with "-debug" and got the same results, but I saw this in $HOME/.gdfuse/default/gdfuse.log
[0.000298] TID=0: Setting up default filesystem...
[0.000335] TID=0: BEGIN: Saving configuration in /home/moses/.gdfuse/default/config
[0.000498] TID=0: END: Saving configuration in /home/moses/.gdfuse/default/config
[0.000543] TID=0: Loading application state from /home/moses/.gdfuse/default/state...done
Current version: 0.6.21
Setting up cache db...done
Setting up CURL...done
[0.001062] TID=0: BEGIN: Saving application state in /home/moses/.gdfuse/default/state
[0.001166] TID=0: END: Saving application state in /home/moses/.gdfuse/default/state
[0.001185] TID=0: BEGIN: Starting web browser with command: xdg-open "https://accounts.google.com/o/oauth2/auth?client_id=564921029129.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fgd-ocaml-auth.appspot.com%2Foauth2callback&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&response_type=code&access_type=offline&approval_prompt=force&state={redacted}"
[0.044670] TID=0: END: Starting web browser with command: xdg-open "https://accounts.google.com/o/oauth2/auth?client_id=564921029129.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fgd-ocaml-auth.appspot.com%2Foauth2callback&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&response_type=code&access_type=offline&approval_prompt=force&state={redacted}"
[0.044703] TID=0: BEGIN: Getting tokens from GAE proxy
[0.598376] TID=0: END: gettokens not found, retrying
[0.615047] TID=0: Retrying (1/8) get_tokens after exception: Not_found
[2.905393] TID=0: END: gettokens not found, retrying
[2.918826] TID=0: Retrying (2/8) get_tokens after exception: Not_found

I've tried xdg-open "https://accounts.google.com" and that works without a problem.  xdg-settings get default-web-browser responds with "chromium.desktop" as expected.

Maybe your issue is similar to this one?

I tried the workaround in that bug report:

sudo chmod 755 /root /root/.config
mkdir /root/.config/chromium
sudo chown -R $USER:$USER /root/.config/chromium

Still the same problem.  Watched it with strace -f, saw execve("/usr/bin/chromium-browser", ["/usr/bin/chromium-browser", "https://accounts.google.com/o/oa"...] but I believe I'm missing the syscalls inside chromium.
Going to try changing my default browser to 'firefox' next... and that worked, it opened a new tab and google prompts me for authenticating with gdfuse: "gdfuse wants to view and manage the files in your Google Drive."
workaround: don't use Chrome/Chromium ?

workaround: don't use Chrome/Chromium ?

I'm using Chrome without issues. Never tried with Chromium.

Same issue for me, executing "xdg-open ..." which i found in gdfuse.log worked.


Fixed order of Windows 8.1 Operating System filter in documentation

Where did AppVeyor go?

Where did AppVeyor go?

Nowhere.
The links are still in README.md#build-status

Where did AppVeyor go?

Nowhere.
The links are still in README.md#build-status

No I mean it didn’t run CI on the PR.

No I mean it didn’t run CI on the PR.

It did; you have to click on the "Current build" or "History" tab to see it.

I think it doesn't have access to this repo to update status. Let's try to fix it? Add me (dblock at dblock dot org) to https://ci.appveyor.com/project/icnocop/dotnetinstaller-ixoxv as an admin?

I replaced the AppVeyor build projects with GitHub Actions as part of PR #146


Add getPlayerThumbnail

This endpoint will return a player's thumbnail when given the user-ID of the player, the thumbnail-type of the thumbnail, and the thumbnail size. This will be useful when displaying information about a user. I'd just like to access these endpoints without a fork. (This is my first pull request, so correct me if I'm wrong.)

Could you add this to the typings file?

Ok I will

Please test your code. It fails $ npm test.

The code now works and passes test, however:

Is this the best endpoint to use? I believe Roblox deprecated the Roblox.com/* endpoints? (We could thumbnails.roblox.com instead?)
The typings file does not contain typings for the required values thumbnailType and size.

Hi, another PR has just been merged which implements this functionality.
Thanks for your PR!


python: use optional-dependencies instead of extras-require

This is the term that PEP 621 uses and it is less likely to be
misspelled.
https://peps.python.org/pep-0621/#dependencies-optional-dependencies
I was never sure if it was extra or extras, or require or
requires and finally committed a mistake in #167405.
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

LGTM, though I'd prefer something shorter like optional or extra.

So let's quickly bikeshed the name further, but I'd like to have it in 22.05, so backporting doesn't become more messy than necessary.

I'm with this if you think it's reasonable. Using the same name as upstream is an advantage.

So be it.

Not sure if it is worth to add some throw alias for it.
Maybe there are already open PRs which use the old syntax, so please pay extra attention to it when reviewing.


How i can use the bottom camera?

try this:
drone.send(at.CONFIG('general:video_channel', 2))  # to switch to bottom camera

drone.send(at.CONFIG('general:video_channel', 1))  # to switch back

plz tell me whether it works!

I tried this, and I confirm it works.


Photutils tutorial notebook #3: aperture photometry

The third notebook tutorial for photutils, demonstrating methods for aperture photometry.
Some preliminary concerns:

Is there anything too important that I left out? The documentation page has a ton of information and examples, and I didn't feel like it would be prudent to include everything. I ended up skipping any demonstrations of alternate methods to define which pixels are in an aperture (i.e. subpixel versus center), more advanced ways of estimating the background within annulus apertures, error estimation, and pixel masking. If one or more of these is really important and should be included, I am happy to do so.
And to directly contradict what I just said: Is this notebook too long? It feels a little unwieldy.
Also, this PR seems to include the deletion of the 2nd notebooks as part of the commit history, and I'm not sure how to undo this. Since the 2nd notebook hasn't been pushed to master yet, I don't know why its deletion appears as a diff. Any ideas?

Tagging @eteq and @Onoddil for review.

Apologies for the weird formatting of this 'review', the commit history for this notebook means I can't just review the diff between this branch and master, so I'm afraid I have to do this review by hand. The line in question will be permalinked and the comments/suggestions etc. will follow it.
In answer to your three questions above:

I don't think any of those things are too important to be put into this notebook, but we might think about adding a notebook 3.5 "advanced aperture photometry" which deals with those things. I don't think we care much about how to define a pixel being in an aperture, but the sigma-clipping of annuli (especially in the XDF where you can see in the cells in the notebook quite a few annuli overlap some other source...) and error (uncertainty!) estimation would be good topics to cover, as I imagine that's exactly the sort of thing a person trying to come to python from IRAF/IDL is going to say: "Oh but IRAF/IDL give me the uncertainties and this doesn't, I'm not going to use it!" and we kinda need to make it as easy as possible to just up and use this stuff.

This notebook feels to be about the right length and does the basics of 'how to make and get a flux from an aperture'

I'm not sure what has happened but I'm having to do this odd not-quite-review because a few of your commits are in master, so I think something odd happened where some commits from a previous branch (presumably notebook 1 which is already in I think?) have got caught up and affected files they shouldn't have and it's revealed them in this branch where you're ahead and behind on commits, but I'm really not sure what has actually happened or how to fix it. For a clean final commit you could just rebase and squash the weird history if the end result is fine and we're happy to lose the commit messages, but I'll leave that to @eteq to advise further...

================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L18
Very minor nitpick, but shouldn't this be the flux of a celestial source? If you think yours is right ignore this, but I think this way reads better, with the flux belonging to the source.
It might also be good to add that the flux is measured on an image (of a CCD/detector) with measures the amount of flux within a region of an astronomical image of defined shape and size.
Very minor change too, I think the flux emitted by a desired source and none of the flux emitted by the surrounding sky might be more correct (with the flux belonging to a source), but I'm not 100% on that so again if you think the original way is right then ignore this.
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L41
As with the previous review, these boxes don't display correctly for me without empty lines between the
 blocks and the text.
"
\n",
"\n",
"**Important:** Before proceeding, please be sure to install or update your [AstroConda](https://astroconda.readthedocs.io) distribution. This notebook may not work properly with older versions of AstroConda\n",
"\n",
 "
"

================
(I'm going to skip the Retrieve data section as it looks fairly copy/paste similar to the previous notebook but I believe there were some comments on the second notebook in this section -- u.ct being u.electron, the units of the colorbar of the plots, and my generic comment about trying to make the masked data black rather than yellow, at least -- so I'm just going to leave a general comment here that any changes suggested there should be applied to all notebook so this section is the same across them all)
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L220
Super minor comment, but I think this should be a previous notebook rather than the previous notebook, as that would then give the flexibility of adding a new notebook at position 2.5 and still have this sentence be fine without editing.
================
(Super generic comment here again about updating all deprecation warnings mentioned in the previous notebook here as well, such as iters to maxiters in sigma_clipped_stats or subpixel=True to centroid_func; any changes made in notebook 2 for the find_peaks section should be reflected in the Creating Apertures section of notebook 3, essentially.)
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L250
Super minor change but perhaps this (and others below) should be print(variable) instead of just variable? It's the same in a notebook I think but a minor change that would allow a user to scoop up all of the cells and make a .py file without the script falling over, and still allow them to verify they got the same result as the notebook did. Not sure whether that's official policy, but I know I prefer working in python scripts rather than notebooks eventually.
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L283
Slightly vague comment here, but I'm not quite sure what we're doing at this point. The only context I've been given is that we can create apertures, then we've searched data for sources and plotted them (fine, this part is a replication of a previous notebook and doesn't need much explanation), but perhaps before the Circular Apertures section there could be a mini paragraph along the lines of "Ok, now that we've found our sources, what can we do with them? We want the calculate their fluxes, so we need to get some apertures inside which we can sum the count rates of all of the pixels. We first try circular apertures as simple examples..." or some such; something that semi-reminds the user what they're doing and why it's useful. There basically just isn't much text from you (as the creator/showrunner of the notebook) in quite a few cells of code, and I really liked in notebook 2 how much explanation there was at each step of the way for the user (also stuff like https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L601 which is a nice little section of explanation!)
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L342
Minor comment, but perhaps this line could be something like aren't any nice round nearby Milky Way stars in this image! to make the obvious point more bluntly that stars are round, galaxies are not? Not much of an issue otherwise just thought it'd be good to go a bit overboard with the "these are galaxies in the XDF!" schtick.
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L373
Similar to above, I think this could perhaps do with a link to notebook 2, with a "see previously for what we did here" to remind the user there's a notebook available to figure out what this thing does and why it's neat. I was briefly thrown as to why the ellipses were all unique until I remembered what detect_sources does myself...
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L378
Similar to above, print(table) instead of table would be good here.
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L435
Similar to the ending of the Circular Aperture section, just before this section begins the elliptical aperture section could do with a closing sentence/paragraph, with something like "as you can see these apertures fit much better, combined with the powerful tools available in the detections toolkit and detect_sources much better than circles for galaxies in XDF..."
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L444
This could perhaps add produce a WCS object from the header of the FITS file containing our image, to make it a bit clearer that the header relates to the picture you're working on.
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L527
Super minor typo: data array on which, not one which.
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L533
Also super minor: should SkyAperture be inside tick quotes, being a photutils thingy?
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L567
Not sure whether this matters much, but I got a warning about the meta attribute being set on the data object but ignored by the function. If the meta flag is used above then I guess this is unavoidable, but otherwise we can likely just remove it to remove a slightly weird not-quite-error message from the notebook.
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L575
Just a reminder that if you change the units from u.ct to u.electron, this cell will also need updating.
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L598
To be consistent with below, the "r = X" entries in this line and the one below it could be tick quoted (r = 3 etc.), as they are in the next section.
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L628
This cell could again be print(phot_table) instead of just phot_table. It could also perhaps have a little sentence lead in of "now that we've (eventually!) made all of our distinct aperture sums for all of our distinct positions, let's see the results:"
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L635
Linked with the above line, this could then also get dressed up a little along the lines of "just printing the results doesn't really tell us much, so let's explore alternative ways to examine these data:" (also these instead of this to be consistent with plural data used throughout the rest of the notebook).
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L653
These plots (and any below, and any above I didn't notice) either just need a unit change to electrons if the swap is made, or to make use of the fancy metadata units plot code Erik mentioned in notebook 2 to print the units of the datacube.
Also not sure if it's particularly necessary but for explicit statement you could add a y label just of 'Number of sources' or something.
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L680
Same issue as above and in notebook 2 where the formatting is shown here unless the lines are separated by line breaks. It also seems that
 doesn't accept ** so those just need removing I think.
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L706
Not 100% sure this is necessary as I'm not sure what level we're quite pitching these notebooks at, but a very brief half-sentence on what an aperture annulus is and how we can use it to background subtract (get average counts in pixels near but outside of object of interest, assumed to be free of source flux, and subtract that flux from all pixels inside the source aperture) might be good just to give a super quick broad context for the start of a new section.
(it seems there kind of is one of these at https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L775 so I'm not sure whether this comment should be changed now but I'm gonna leave it as it was and just suggest perhaps that this latter section is moved up to the previous part I wanted this context in)
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L779
Should background value be something like the count rate of the background or the background count rate? value sounds a bit vague, could do with being a little bit more specific I think. Are you worried it gets confused with count rate within each annulus in the first step?
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L805
Here and cell two down should maybe be print(variable) instead of just variable again.
================
https://github.com/eteq/csi-stsci-notebooks/blob/fc0248b0a6fa504b988fa4aa0c352651fa2da52a/photutils/03_photutils_aperture_photometry.ipynb#L814
Might be good here to have a brief little sentence about the fact that because these data are already background subtracted we'd hope that the background is tiny and oh look it is, just to reassure users that this function does have use honest and that 0.0007 ct/s is fine as a value to get... I also think this should possibly have units u.electron / u.s / u.pixel (or u.ct) as that makes it clear that this is the count rate in a single pixel.
================
Hopefully this review is ok and makes sense - let me know if you need any clarification of anywhere I've tried to link to or not formatted right, etc.

Thanks so much for this great feedback, @Onoddil!
Again, per the new plan for finishing up these notebooks, I am going to close this PR - any future discussion will take place on the spacetelescope/notebooks repo, on the new PR thread: spacetelescope/notebooks#101
I have implemented almost all your thoughts, though, and I very much appreciate your attention to detail. I am choosing not to use some of your suggestions though, including:

The places where I don't explicitly put the Astropy tables in a print() statement is because printing an Astropy table returns a hunk of ASCII text, while just returning the table renders a lovely HTML-formatted table, so I'm planning to keep those the way they are.
I couldn't find a particularly good way to move the bulleted list explanation of aperture correction up to the top of that section, but I did reword a few things to hopefully make it flow better. Let me know if that still isn't to your liking.
I tried to implement your last comment about specifying the background count rate units as u.electron / u.s / **u.pixel**, but that quickly became very messy, since the aperture area should really be in (u.pixel)**2, so I am thinking I will just leave it as it is as well.


Notes bugfix (task #4481)

Fixes broken functionality for changing note type and visibility, which was introduced since AdminLTE version update.

Codecov Report

Merging #45 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #45   +/-   ##
=========================================
  Coverage     65.16%   65.16%
  Complexity       22       22
=========================================
  Files             3        3
  Lines           155      155
=========================================
  Hits            101      101
  Misses           54       54

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b451724...b2df35d. Read the comment docs.


java.lang.NullPointerException:Attempt to invoke virtual method 'void android.widget.TextView.setVisibility(int)' on a null object reference

Is the binding control successful in some of the destruction reuse of controls because the reasons for the use of the same ID binding in different layouts

You're either accessing a view before calling bind (or after calling unbind) or it's marked as @Nullable and you're not null checking it. Or you're forgetting to call bind altogether.

The null pointer appears occasionally, with different models. The binding view is bound in the OnCreat method, and the call to the bound view in some callbacks is the occasional null-pointer, not every time.


chore: Remove release please & bump versions

I want to move away from release-please because:

It continues trying to release wnfs-bench, even though it shouldn't. (We've tried removing wnfs-bench from all configuration JSONs, but release-please just adds it back, presumably because it reads it from the history somewhere)
It doesn't handle new crates in the workspace well at all. (Between removing wnfs-namefilter and adding wnfs-nameaccumulator it just got totally confused and we had to manually publish the second release of wnfs-nameaccumulator. This also caused all other crates to not release properly, since they depend on wnfs-nameaccumulator...)
Now that we've done one manual release, it's gotten completely confused. (See #344 and the CHANGELOGs it generated)
While putting together this PR, I realized that the CHANGELOG formatting got messed up at some point for the wnfs-hamt crate.

So this PR simply removes the workflow.
I also bumped the versions in preparation for a release & added CHANGELOGs.

Yep. Back to manual :/


Question: implement as package provider?

This is just a question: wouldn't it be more natural to implement this as a package provider?
package { 'Foo::Bar':
  ensure   => latest,
  provider => 'cpan',
}

@faxm0dem pull requests are still welcomed

:-) I'm still interested in doing this thanks for reminding me

sorry, time doesn't permit this as of now. I'll close this and open a PR eventually


Feature - activities, contacts, sms, styles

Needs to be merged before this one: https://github.com/codeAdrian/homeBound/pull/3

Activities functionality
Contacts functionality
SMS functionality
Node Server setup
Twilio setup (see .env.sample)
Base styles
Login, signup, survey styles
Misc improvements

Can you actually rebase this over the other PR, or merge that one and rebase, considering that I see many of the same file changes here, making it a bit more difficult to properly review.


error setting pythonVirtualEnv parameter

Hi everyone. Im struggling setting up this extension.
Well im on Windows using VS Code connected to Linux Development Server throw SSH

So As you can see in the screenshot python extension show python global environment wich is /bin/python3
But when i put this in parameter still have error
pythonVirtualEnv not set in settings. Please add it
Anyone can help me? Thanks in advance

@janderbenitez the virtualenv path must be absolute path (in Windows you need to use Windows path forma 'C:/path .., I guess). I use this extension in a local virtualenv and work.

@janderbenitez the virtualenv path must be absolute path (in Windows you need to use Windows path forma 'C:/path .., I guess). I use this extension in a local virtualenv and work.

Hi mstuttgart thanks for reply. Well im on windows but server is linux. Remember wich i connect throw SSH to Ubuntu Server 22.04 there is the app dont use docker is on premise.

If i surf to /bin i could find python folders wich i set in extension api but i doesnt work each one


Allow assigning arbritrary attributes to nodes, edges and graphs

Allow arbritrary attributes to be set on nodes and edges
Add graph attributes

pls resolve conflicts

Rebased onto current develop

Hey @przygienda,
First of all, thank you for your time and for your contributions to the open-source community.
I am a user of this project, and believe that this would be a valuable addition to the project, and would be grateful if you could consider merging this.


[BUG]

Minimal Code To Reproduce
import pandas as pd
from prefect import task, flow
from prefect_fugue import fugue_engine
from fugue import fsql

@task
def load_data():
    return pd.read_parquet("https://d37ci6vzurychx.cloudfront.net/trip-data/green_tripdata_2022-01.parquet")

@flow
def run_sql(top, engine):
    data = load_data()
    with fugue_engine(engine):
        fsql("""
        SELECT PULocationID, COUNT(*) AS ct FROM df
        GROUP BY 1 ORDER BY 2 DESC LIMIT 2
        PRINT
        """, data)

run_sql(1, "spark")

Describe the bug
"ImportError: ord() expected string of length 1, but int found. Please try to install the package by pip install fugue[sql]."

Full log:
/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue_sql/__init__.py:8: UserWarning: fsql and FugueSQLWorkflow now should be imported directly from fugue, fugue_sql will be removed in 0.9.0
  warnings.warn(
02:17:10.612 | INFO    | prefect.engine - Created flow run 'finicky-cat' for flow 'run-sql'
02:17:10.752 | INFO    | Flow run 'finicky-cat' - Created task run 'load_data-0' for task 'load_data'
02:17:10.754 | INFO    | Flow run 'finicky-cat' - Executing 'load_data-0' immediately...
02:17:12.186 | INFO    | Task run 'load_data-0' - Finished in state Completed()
23/10/03 02:17:15 WARN Utils: Your hostname, WDX5CG01992LV0 resolves to a loopback address: <IP_ADDRESS>; using <IP_ADDRESS> instead (on interface eth0)
23/10/03 02:17:15 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/10/03 02:17:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
02:17:19.667 | ERROR   | Flow run 'finicky-cat' - Encountered exception during execution:
Traceback (most recent call last):
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue/_utils/misc.py", line 20, in import_or_throw
    return __import__(package_name)
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue_sql_antlr/__init__.py", line 2, in <module>
    from fugue_sql_antlr.constants import (
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue_sql_antlr/constants.py", line 1, in <module>
    from fugue_sql_antlr._parser.sa_fugue_sql import USE_CPP_IMPLEMENTATION
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue_sql_antlr/_parser/sa_fugue_sql.py", line 12, in <module>
    from .fugue_sqlParser import fugue_sqlParser
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue_sql_antlr/_parser/fugue_sqlParser.py", line 1720, in <module>
    class fugue_sqlParser ( Parser ):
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue_sql_antlr/_parser/fugue_sqlParser.py", line 1724, in fugue_sqlParser
    atn = ATNDeserializer().deserialize(serializedATN())
  File "/home/user/.local/lib/python3.10/site-packages/antlr4/atn/ATNDeserializer.py", line 61, in deserialize
    self.reset(data)
  File "/home/user/.local/lib/python3.10/site-packages/antlr4/atn/ATNDeserializer.py", line 91, in reset
    temp = [ adjust(c) for c in data ]
  File "/home/user/.local/lib/python3.10/site-packages/antlr4/atn/ATNDeserializer.py", line 91, in <listcomp>
    temp = [ adjust(c) for c in data ]
  File "/home/user/.local/lib/python3.10/site-packages/antlr4/atn/ATNDeserializer.py", line 89, in adjust
    v = ord(c)
TypeError: ord() expected string of length 1, but int found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/engine.py", line 829, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 291, in aresult
    return await asyncio.wrap_future(self.future)
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 315, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/home/user/fugue_ibis_experiment/fuguecreator.py", line 16, in run_sql
    fsql("""
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue/sql/api.py", line 252, in fugue_sql_flow
    dag = _build_dag(
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue/sql/api.py", line 284, in _build_dag
    dag._sql(query, global_vars, local_vars, *args, **kwargs)
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue/sql/workflow.py", line 39, in _sql
    FugueSQLParser = import_fsql_dependency("fugue_sql_antlr").FugueSQLParser
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue/_utils/misc.py", line 26, in import_fsql_dependency
    return import_or_throw(
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue/_utils/misc.py", line 22, in import_or_throw
    raise ImportError(str(e) + ". " + message)
ImportError: ord() expected string of length 1, but int found. Please try to install the package by `pip install fugue[sql]`.
02:17:19.757 | ERROR   | Flow run 'finicky-cat' - Finished in state Failed('Flow run encountered an exception. ImportError: ord() expected string of length 1, but int found. Please try to install the package by `pip install fugue[sql]`.')
Traceback (most recent call last):
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue/_utils/misc.py", line 20, in import_or_throw
    return __import__(package_name)
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue_sql_antlr/__init__.py", line 2, in <module>
    from fugue_sql_antlr.constants import (
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue_sql_antlr/constants.py", line 1, in <module>
    from fugue_sql_antlr._parser.sa_fugue_sql import USE_CPP_IMPLEMENTATION
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue_sql_antlr/_parser/sa_fugue_sql.py", line 12, in <module>
    from .fugue_sqlParser import fugue_sqlParser
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue_sql_antlr/_parser/fugue_sqlParser.py", line 1720, in <module>
    class fugue_sqlParser ( Parser ):
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue_sql_antlr/_parser/fugue_sqlParser.py", line 1724, in fugue_sqlParser
    atn = ATNDeserializer().deserialize(serializedATN())
  File "/home/user/.local/lib/python3.10/site-packages/antlr4/atn/ATNDeserializer.py", line 61, in deserialize
    self.reset(data)
  File "/home/user/.local/lib/python3.10/site-packages/antlr4/atn/ATNDeserializer.py", line 91, in reset
    temp = [ adjust(c) for c in data ]
  File "/home/user/.local/lib/python3.10/site-packages/antlr4/atn/ATNDeserializer.py", line 91, in <listcomp>
    temp = [ adjust(c) for c in data ]
  File "/home/user/.local/lib/python3.10/site-packages/antlr4/atn/ATNDeserializer.py", line 89, in adjust
    v = ord(c)
TypeError: ord() expected string of length 1, but int found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/fugue_ibis_experiment/fuguecreator.py", line 23, in <module>
    run_sql(1, "spark")
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/flows.py", line 758, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/engine.py", line 271, in enter_flow_run_engine_from_flow_call
    retval = from_sync.wait_for_call_in_loop_thread(
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/_internal/concurrency/api.py", line 243, in wait_for_call_in_loop_thread
    return call.result()
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 282, in result
    return self.future.result(timeout=timeout)
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 168, in result
    return self.__get_result()
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 345, in _run_async
    result = await coro
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/engine.py", line 374, in create_then_begin_flow_run
    return await state.result(fetch=True)
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/states.py", line 91, in _get_state_result
    raise await get_state_exception(state)
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/engine.py", line 829, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 291, in aresult
    return await asyncio.wrap_future(self.future)
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 315, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/home/user/fugue_ibis_experiment/fuguecreator.py", line 16, in run_sql
    fsql("""
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue/sql/api.py", line 252, in fugue_sql_flow
    dag = _build_dag(
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue/sql/api.py", line 284, in _build_dag
    dag._sql(query, global_vars, local_vars, *args, **kwargs)
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue/sql/workflow.py", line 39, in _sql
    FugueSQLParser = import_fsql_dependency("fugue_sql_antlr").FugueSQLParser
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue/_utils/misc.py", line 26, in import_fsql_dependency
    return import_or_throw(
  File "/home/user/fugue_ibis_experiment/.venv/lib/python3.10/site-packages/fugue/_utils/misc.py", line 22, in import_or_throw
    raise ImportError(str(e) + ". " + message)
ImportError: ord() expected string of length 1, but int found. Please try to install the package by `pip install fugue[sql]`.

Already reinstalled, but the error keeps on happening
Expected behavior
Basically the same as the tutorial: https://fugue-tutorials.readthedocs.io/tutorials/integrations/ecosystem/prefect.html#running-sql-on-any-spark-dask-and-duckdb
Environment (please complete the following information):

Backend: pandas/dask/ray? spark (but none of the others are working too)
Backend version:
Python version: 3.10.12
OS: wsl2

packages installed:
[project]
dependencies = [
    "prefect-fugue>=0.0.1",
    "pandas>=2.1.0",
    "pyspark>=3.4.0",
    "fugue[sql]>=0.8.6",
]

Pip freeze:
adagio==0.2.4
aiohttp==3.8.5
aiosignal==1.3.1
aiosqlite==0.19.0
alembic==1.12.0
antlr4-python3-runtime==4.9.3
anyconfig==0.13.0
anyio==3.7.1
appdirs==1.4.4
apprise==1.5.0
arrow==1.2.3
asgi-lifespan==2.1.0
astroid==2.13.5
asttokens==2.4.0
async-timeout==4.0.3
asyncpg==0.28.0
atpublic==4.0
attrs==23.1.0
backcall==0.2.0
binaryornot==0.4.4
black==23.3.0
bleach==6.0.0
blinker==1.6.2
boto3==1.26.32
botocore==1.29.165
build==1.0.3
CacheControl==0.13.1
cachetools==5.3.1
certifi==2023.7.22
cffi==1.16.0
cfgv==3.3.1
charset-normalizer==2.1.1
click==8.1.7
cloudpickle==2.2.1
colorama==0.4.6
contourpy==1.1.1
cookiecutter==2.1.1
coolname==2.2.0
coverage==7.2.7
croniter==1.4.1
cryptography==41.0.4
cx-Oracle==8.3.0
cycler==0.11.0
databricks-cli==0.17.8
dateparser==1.1.8
decorator==5.1.1
delta-spark==2.1.0
diff-cover==7.7.0
dill==0.3.7
distlib==0.3.7
docker==6.1.3
dynaconf==3.2.3
entrypoints==0.4
et-xmlfile==1.1.0
exceptiongroup==1.1.2
executing==1.2.0
fastapi==0.103.2
filelock==3.12.2
findpython==0.4.0
findspark==2.0.1
flake8==5.0.4
Flask==2.3.3
fonttools==4.42.1
fs==2.4.16
fsspec==2023.9.2
fugue==0.8.6
fugue-sql-antlr==0.1.7
gitdb==4.0.10
GitPython==3.1.37
google-auth==2.23.2
graphviz==0.20.1
greenlet==2.0.2
griffe==0.36.4
gunicorn==21.2.0
h11==0.14.0
h2==4.1.0
hpack==4.0.0
httpcore==0.18.0
httpx==0.25.0
hyperframe==6.0.1
identify==2.5.26
idna==3.4
importlib-resources==6.1.0
iniconfig==2.0.0
installer==0.7.0
ipython==8.15.0
isort==5.12.0
itsdangerous==2.1.2
JayDeBeApi==1.2.3
jedi==0.19.0
Jinja2==3.1.2
jinja2-time==0.2.0
jmespath==1.0.1
joblib==1.3.2
JPype1==1.4.0
jsonpatch==1.33
jsonpointer==2.4
jsonschema==4.19.1
jsonschema-specifications==2023.7.1
kedro==0.18.13
kiwisolver==1.4.5
kubernetes==28.1.0
lazy-object-proxy==1.9.0
Mako==1.2.4
Markdown==3.4.4
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib==3.8.0
matplotlib-inline==0.1.6
mccabe==0.7.0
mdurl==0.1.2
mlflow==2.7.1
more-itertools==10.1.0
msgpack==1.0.7
mypy-extensions==1.0.0
nodeenv==1.8.0
numpy==1.23.0
oauthlib==3.2.2
omegaconf==2.3.0
openpyxl==3.0.10
oracledb==1.4.1
orjson==3.9.7
packaging==23.1
pandas==1.4.3
parse==1.19.1
parso==0.8.3
pathspec==0.11.2
pdm==2.9.3
pendulum==2.1.2
pickleshare==0.7.5
Pillow==10.0.1
pip-tools==7.3.0
platformdirs==3.10.0
pluggy==1.2.0
pre-commit==2.21.0
prefect==2.13.4
prefect-fugue==0.0.1
prompt-toolkit==3.0.39
protobuf==4.24.3
pure-eval==0.2.2
py4j==<IP_ADDRESS>
pyarrow==13.0.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycodestyle==2.9.1
pycparser==2.21
pydantic==1.10.13
pyflakes==2.5.0
Pygments==2.16.1
pylint==2.15.9
pyproject_hooks==1.0.0
pyspark==3.5.0
pytest==7.4.0
pytest-cov==3.0.0
pytest-mock==3.8.2
pytest-randomly==3.12.0
pytest-spark==0.6.0
python-benedict==0.32.0
python-dateutil==2.8.2
python-dotenv==1.0.0
python-fsutil==0.10.0
python-slugify==8.0.1
pytoolconfig==1.2.5
pytz==2023.3.post1
pytzdata==2020.1
PyYAML==6.0
qpd==0.4.4
querystring-parser==1.2.4
readchar==4.0.5
referencing==0.30.2
regex==2023.8.8
requests==2.28.1
requests-oauthlib==1.3.1
requests-toolbelt==1.0.0
resolvelib==1.0.1
rich==13.5.3
rope==1.9.0
rpds-py==0.10.3
rsa==4.9
ruamel.yaml==0.17.33
ruamel.yaml.clib==0.2.7
s3transfer==0.6.1
scikit-learn==1.3.1
scipy==1.11.2
shellingham==1.5.3
six==1.16.0
smmap==5.0.1
sniffio==1.3.0
SQLAlchemy==2.0.21
sqlfluff==2.3.2
sqlglot==18.7.0
sqlparse==0.4.2
sqlparser==0.0.9
stack-data==0.6.2
starlette==0.27.0
tabulate==0.9.0
tblib==2.0.0
text-unidecode==1.3
threadpoolctl==3.2.0
toml==0.10.2
tomli==2.0.1
tomlkit==0.12.1
toposort==1.10
tqdm==4.66.0
traitlets==5.10.0
triad==0.9.1
truststore==0.8.0
typer==0.9.0
typing_extensions==4.7.1
tzdata==2023.3
tzlocal==5.0.1
unearth==0.11.0
urllib3==1.26.16
uvicorn==0.23.2
virtualenv==20.24.2
wcwidth==0.2.6
webencodings==0.5.1
websocket-client==1.6.3
websockets==11.0.3
Werkzeug==2.3.7
wrapt==1.15.0

@guilhermedelyra thanks so much for reporting!
Acutally prefect-fugue needs refactoring because it was designed before a major refactoring of Fugue.
After the fix on the prefect-fugue side, the experience should be even simpler:
import pandas as pd
import fugue.api as fa
from prefect import task, flow

@task
def load_data():
    return pd.read_parquet("https://d37ci6vzurychx.cloudfront.net/trip-data/green_tripdata_2022-01.parquet")

@flow
def run_sql(top, engine):
    data = load_data()
    with fa.engine_context(engine):
        fa.fugue_sql_flow("""
        SELECT PULocationID, COUNT(*) AS ct FROM df
        GROUP BY 1 ORDER BY 2 DESC LIMIT 2
        PRINT
        """, data)

run_sql(1, "spark") # your current spark session (not recommended)
run_sql(1, "prefect_fugue/<spark_block>") # using the config in your prefect block to connect to remote spark/databricks

that's really nice to know 🙂
Is there anything I can do to help?

@guilhermedelyra for this one, this is a long due change, and I just made it. Thanks for raising this problem.
https://github.com/fugue-project/prefect-fugue/pull/9
If you are interested in contributing to either the prefect-fugue or the fugue project, please let us know. We can discuss on slack channel.
Thanks!

@guilhermedelyra we have released prefect-fugue 0.0.3, and you can find instruction here: https://github.com/fugue-project/prefect-fugue/pull/10
Please let us know if you have any question or feedbacks.
Thanks

thank you very much! 🙂
I'll be in contact any time soon


Add Meson build language support

Add a Meson language lexer

update the mappings

Add meson to the list of supported languages

Can you please add an example file and a golden pytest output for it?

Can you please add an example file and a golden pytest output for it?
Well that sure turned up some bugs and missing pieces!
The example meson.build is nonsense but valid meson syntax, and covers most of
the language that I can see.

I also expanded the list of functions to the complete list from the reference
manual and also expanded the magic keywords.

--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/pygments/pygments/pull/1823#issuecomment-850136990

Merged for the next release, thanks!


Fix the SCM connection URL in pom.xml

The "connection" should be an URL valid for read-only access and has to follow
the syntax as described at https://maven.apache.org/pom.html#SCM.

@ceki, any chance go get this trivial fix merged before the next release?


No implementation found for method getArtwork on channel boaventura.com.devel.br.flutteraudioquery

Hi,
Can you fix the unhandled exception?
The app can not be tested, basically. It freezes everything. It keeps crashing, no matter how many breakpoints I remove from vscode. Can you handle that exception, please?
My code is this:
Future getArtWork({SongInfo song}) async {
try {
return audioQuery.getArtwork(
type: ResourceType.SONG, id: song.id, size: Size(100, 100));
} catch (e) {
print("Error loading Artwork");
return null;
}
}
The error is the same as the other open issue. I am opening another issue because I would like to you this plugin and it is not working.

I just used this instead:
import "dart.io" as IO;
CircleAvatar(
backgroundImage:
FileImage(IO.File(song?.albumArtwork)),
)


$vssHeight and other computed vars do not work

... but the functions themselves work when called directly e.g. this.getScreenWidth() 👍

Hmm, that's strange. It's confirmed working in the latest demo. Perhaps you could try upgrading to the latest version.
Closing this for now, will reopen if needed. Thank you.


Skip if not in praw-dev/praw repo

Secrets are not copied to forks, so the program won't run anywhere else.

Thanks!


dnsproxy: Transparent DNS Proxy i/o timeout with node-local-dns

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
We recently upgraded from Cilium v1.14.5 to v1.14.12. Very quickly after, we started noticing some DNS requests getting dropped:
Failed to resolve 'eng-my-svc-internal-microservice.default.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [default.svc.cluster.local, svc.cluster.local, cluster.local]

We were able to correlate these issues with the following messages:
time="2024-06-13T16:37:36Z" level=warning msg="Timeout waiting for response to forwarded proxied DNS lookup" DNSRequestID=50346 dnsName=eng-my-svc-internal.microservice.default.svc.cluster.local. endpointID=1372 error="read udp <IP_ADDRESS>:58491-><IP_ADDRESS>:53: i/o timeout" identity=13826 ipAddr="<IP_ADDRESS>:38680" subsys=fqdn/dnsproxy

After verifying that the issue wasn't caused by resource utilization on our daemonset nodelocaldns / CoreDNS, nor was it caused by a client issue (i.e. we had this issue across Python/Java/Ruby services).
This issue causes sporadic resolution failures, especially under load (our dev/staging environments show much fewer occurrences).
We were able to verify that the issue occurs starting from v1.14.6 to v1.14.12.
This allowed us to isolate this change:

Cilium DNS proxy can now use the original pod's address as the source address towards the DNS servers (--dnsproxy-enable-transparent-mode). (Backport PR https://github.com/cilium/cilium/pull/30213, Upstream PR https://github.com/cilium/cilium/pull/29239, @jrajahalme)

Our Helm install had indeed enabled transparent mode as we upgraded Cilium from v1.12.5 to v1.12.12. Disabling dnsproxy-enable-transparent-mode results in the DNS resolution issues to completely stop.
Cilium Version
v1.14.12 (started in v1.14.6)
Kernel Version
Linux ip-10-212-165-205.eu-west-1.compute.internal 5.10.217-205.860.amzn2.x86_64 #1 SMP Tue May 21 16:52:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Kubernetes Version
v1.27
Regression
No response
Sysdump
From the Slack thread: https://cilium.slack.com/files/U01T2KHTE8P/F077VK7DBCL/cilium-sysdump-20240612-172110.zip
Relevant log output
time="2024-06-12T12:43:51Z" level=debug msg="Handling DNS query from endpoint" DNSRequestID=43145 dnsName=my-service.eu-dev.my-domain.com. ipAddr="<IP_ADDRESS>:50497" subsys=fqdn/dnsproxy
time="2024-06-12T12:43:51Z" level=debug msg="Found target server to of DNS request secID {ID:16777220 Source:custom-resource _:[] shadowed:false createdFromMetadata:false}" DNSRequestID=43145 dnsName=my-service.eu-dev.my-domain.com. endpointID=1265 identity=225474 ipAddr="<IP_ADDRESS>:50497" server="<IP_ADDRESS>:53" subsys=fqdn/dnsproxy
time="2024-06-12T12:43:51Z" level=debug msg="Forwarding DNS request for a name that is allowed" DNSRequestID=43145 dnsName=my-service.eu-dev.my-domain.com. endpointID=1265 identity=225474 ipAddr="<IP_ADDRESS>:50497" subsys=fqdn/dnsproxy
time="2024-06-12T12:43:52Z" level=debug msg="Handling DNS query from endpoint" DNSRequestID=43145 dnsName=my-service.eu-dev.my-domain.com. ipAddr="<IP_ADDRESS>:50497" subsys=fqdn/dnsproxy
time="2024-06-12T12:43:52Z" level=debug msg="Found target server to of DNS request secID {ID:16777220 Source:custom-resource _:[] shadowed:false createdFromMetadata:false}" DNSRequestID=43145 dnsName=my-service.eu-dev.my-domain.com. endpointID=1265 identity=225474 ipAddr="<IP_ADDRESS>:50497" server="<IP_ADDRESS>:53" subsys=fqdn/dnsproxy
time="2024-06-12T12:43:52Z" level=debug msg="Forwarding DNS request for a name that is allowed" DNSRequestID=43145 dnsName=my-service.eu-dev.my-domain.com. endpointID=1265 identity=225474 ipAddr="<IP_ADDRESS>:50497" subsys=fqdn/dnsproxy
time="2024-06-12T12:43:52Z" level=debug msg="Received DNS response to proxied lookup" DNSRequestID=43145 dnsName=my-service.eu-dev.my-domain.com. endpointID=1265 identity=225474 ipAddr="<IP_ADDRESS>:50497" resp=";; opcode: QUERY, status: NOERROR, id: 20903;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1;; OPT PSEUDOSECTION:; EDNS: version 0; flags: ; udp: 1200;; QUESTION SECTION:;my-service.eu-dev.my-domain.com.\tIN\t A;; ANSWER SECTION:my-service.eu-dev.my-domain.com.\t5\tIN\tCNAME\tmy-service-cluster.cluster-xxxx.eu-west-1.rds.amazonaws.com.my-service-cluster.cluster-xxxx.eu-west-1.rds.amazonaws.com.\t5\tIN\tCNAME\tmy-service-cluster.cluster-xxxx.eu-west-1.rds.amazonaws.com.my-service-cluster.cluster-xxxx.eu-west-1.rds.amazonaws.com.\t5\tIN\tA\t<IP_ADDRESS>" subsys=fqdn/dnsproxy
time="2024-06-12T12:43:52Z" level=debug msg="Notifying with DNS response to original DNS query" DNSRequestID=43145 dnsName=my-service.eu-dev.my-domain.com. endpointID=1265 identity=225474 ipAddr="<IP_ADDRESS>:50497" subsys=fqdn/dnsproxy
time="2024-06-12T12:43:52Z" level=debug msg="Responding to original DNS query" DNSRequestID=43145 dnsName=my-service.eu-dev.my-domain.com. endpointID=1265 identity=225474 ipAddr="<IP_ADDRESS>:50497" subsys=fqdn/dnsproxy
time="2024-06-12T12:44:01Z" level=warning msg="Timeout waiting for response to forwarded proxied DNS lookup" DNSRequestID=43145 dnsName=my-service.eu-dev.my-domain.com. endpointID=1265 error="read udp <IP_ADDRESS>:32769-><IP_ADDRESS>:53: i/o timeout" identity=225474 ipAddr="<IP_ADDRESS>:50497" subsys=fqdn/dnsproxy

Anything else?
 Please add yourself to the Users doc

Code of Conduct

[X] I agree to follow this project's Code of Conduct

But not for every

Interesting. Another side-effect of the dns proxy source address spoofing.
For us that feature broke autopath: #20191
Good to know that dnsproxy-enable-transparent-mode: false could be an alternative workaround.

But not for everyone:
time="2024-06-17T15:28:19Z" level=fatal msg="IPSec requires DNS proxy transparent mode to be enabled (--dnsproxy-enable-transparent-mode=\"true\")" subsys=daemon

Hey @mantoine96, really appreciate you setting up a repo for reproducing - that was super helpful, thank you! I was able to reproduce the behavior you're seeing, but only temporarily due to the identity of node-local-dns changing:
After deployment, node-local-dns is entered into the ipcache and lxc map with the host identity:
root@kind-worker:/home/cilium# cilium bpf ipcache list | grep <IP_ADDRESS>
<IP_ADDRESS>/32        identity=1 encryptkey=0 tunnelendpoint=<IP_ADDRESS>

root@kind-worker:/home/cilium# hubble observe -f --pod default/tiefighter
Jun 24 23:22:50.538: default/tiefighter:53895 (ID:2012) -> <IP_ADDRESS>:53 (host) dns-request proxy FORWARDED (DNS Query deathstar.default.svc.cluster.local. A)
Jun 24 23:22:50.540: default/tiefighter:53895 (ID:2012) <- <IP_ADDRESS>:53 (host) dns-response proxy FORWARDED (DNS Answer "<IP_ADDRESS>" TTL: 6 (Proxy deathstar.default.svc.cluster.local. A))
Jun 24 23:22:50.540: default/tiefighter:53895 (ID:2012) <- <IP_ADDRESS>:53 (host) to-endpoint FORWARDED (UDP)

Once cilium-agent is restarted, the ip is considered a stale host entry and is removed from the ipcache and lxc map. From here on, node-local-dns is considered egress to world, which breaks the rule-tiefighter policy:
ks logs cilium-n8hl7 -f | grep <IP_ADDRESS>
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
time="2024-06-25T00:06:11Z" level=debug msg="Removed outdated host IP <IP_ADDRESS> from endpoint map" subsys=daemon

root@kind-worker:/home/cilium# cilium bpf ipcache list | grep 'identity=1 '
<IP_ADDRESS>/32           identity=1 encryptkey=0 tunnelendpoint=<IP_ADDRESS>
<IP_ADDRESS>/32           identity=1 encryptkey=0 tunnelendpoint=<IP_ADDRESS>
fc00:c111::3/128        identity=1 encryptkey=0 tunnelendpoint=<IP_ADDRESS>

root@kind-worker:/home/cilium# hubble observe -f --pod default/tiefighter
Jun 24 23:38:35.928: default/tiefighter:49425 (ID:2012) <> <IP_ADDRESS>:53 (world) policy-verdict:none EGRESS DENIED (UDP)
Jun 24 23:38:35.928: default/tiefighter:49425 (ID:2012) <> <IP_ADDRESS>:53 (world) Policy denied DROPPED (UDP)
Jun 24 23:38:35.928: default/tiefighter:49425 (ID:2012) <> <IP_ADDRESS>:53 (world) policy-verdict:none EGRESS DENIED (UDP)
Jun 24 23:38:35.928: default/tiefighter:49425 (ID:2012) <> <IP_ADDRESS>:53 (world) Policy denied DROPPED (UDP)
Jun 24 23:38:36.930: default/tiefighter:49425 (ID:2012) <> <IP_ADDRESS>:53 (world) policy-verdict:none EGRESS DENIED (UDP)
Jun 24 23:38:36.930: default/tiefighter:49425 (ID:2012) <> <IP_ADDRESS>:53 (world) Policy denied DROPPED (UDP)

To let tiefighter talk to node-local-dns, I had to modify the policy to use a CIDR (<IP_ADDRESS>/16, since node-local-dns uses a link-local IP). Since this IP is considered external to the cluster, this conditional evaluates to false and the key is left empty. With an empty key, the shared config is not actually shared: https://github.com/cilium/dns/blob/af788769786ae96ebbb5506625746bdf0a654cb6/shared_client.go#L46
This is relevant because while stress testing with siege in your repro environment, the dnsproxy queries began to timeout once all of the ports were exhausted, which is due to the requests being transparently proxied using the original source address. Because siege is using multiple clients, a new connection on a new port is initiated for each transaction and quickly consumes all available ports.
We're not currently checking for link-local unicast addresses here, but doing so might cause these requests to skip the transparent mode, and seems like a case we'd probably want to cover in general regardless.
With all that said, the configuration you're using is not the recommended way of deploying node-local-dns with Cilium. We have a guide that outlines deploying it using a LocalRedirectPolicy and running node-local-dns in the pod network, which would allow you to avoid using a custom DNS config on all deployed pods. You can find that guide here, but I think the simpler option here is to just keep the feature disabled.


Device name can be null in some circumstances

We didn't reproduce this but based on this crash and similar guards elsewhere in the code this seems a reasonable fix.
https://play.google.com/console/u/0/developers/4695663865375895527/app/4975736116501095047/vitals/crashes/e9abff6694fe9443f6d3423bb5f7b44b/details?days=28&versionCode=57&isUserPerceived=true

I can't see the link (presumably crash report). The change looks good.

Sorry Martin. Trace from the crash report for context (Android 14):
Exception java.lang.RuntimeException:
  at android.app.LoadedApk$ReceiverDispatcher$Args.lambda$getRunnable$0 (LoadedApk.java:1876)
  at android.app.LoadedApk$ReceiverDispatcher$Args.$r8$lambda$gDuJqgxY6Zb-ifyeubKeivTLAwk
  at android.app.LoadedApk$ReceiverDispatcher$Args$$ExternalSyntheticLambda0.run
  at android.os.Handler.handleCallback (Handler.java:958)
  at android.os.Handler.dispatchMessage (Handler.java:99)
  at android.os.Looper.loopOnce (Looper.java:222)
  at android.os.Looper.loop (Looper.java:314)
  at android.app.ActivityThread.main (ActivityThread.java:8706)
  at java.lang.reflect.Method.invoke
  at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:565)
  at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:1081)
Caused by java.lang.NullPointerException: Attempt to invoke virtual method 'boolean java.lang.String.isEmpty()' on a null object reference
  at com.samsung.microbit.utils.BLEPair$9.onReceive (BLEPair.java:646)
  at android.app.LoadedApk$ReceiverDispatcher$Args.lambda$getRunnable$0 (LoadedApk.java:1866)

That proves it can happen! I haven't seen it on Android 8.1, 12, 13 or 14. If I had been thinking, I might have checked addr and action too.


3 high severity Chromium vulnerabilities in VSCode 1.81 - patched in Electron 22.3.20

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.81.0
OS Version: Windows 10 x64

Steps to Reproduce:
The latest VSCode July 1.81 includes Electron 22.3.18 which has 3 high severity vulnerabilities in Chromium.
https://github.com/advisories/GHSA-6f46-9vvr-v3j5
https://github.com/advisories/GHSA-fxgf-5cm8-2f8q
https://github.com/advisories/GHSA-2gmm-4f9j-mw4p
These have been patched/backported in Electron 22.3.20 https://github.com/electron/electron/releases/tag/v22.3.20
@deepak1556 Would you please include these fixes in the new July Recovery?

Thanks for bringing this up, we don't have internal builds available yet for 22.3.20 so it won't make the cut for 1.81.1 but I will try to get it in for 1.81.2 instead.

Out of the 3 issues the only relevant one for VSCode is https://github.com/advisories/GHSA-6f46-9vvr-v3j5

Thanks for bringing this up, we don't have internal builds available yet for 22.3.20 so it won't make the cut for 1.81.1 but I will try to get it in for 1.81.2 instead.

@deepak1556 That would be good, but will there even be a 1.81.2? I don't see any sign of one being created in milestones https://github.com/microsoft/vscode/milestones

Yeah we don't have plans for 1.81.2 at this time, after looking at the CVE we have decided it to not be a candidate. Thanks again for raising the issue!

@deepak1556 Please reconsider incorporating the latest Electron 22.3.22 backports into VSCode 1.81.2. Several high severity vulnerabilities have been corrected.
https://github.com/electron/electron/releases/tag/v22.3.22


New docker-coompose.yml

I generated a new docker config with sail. It worked for me, but I have no idea if this is general enough.
@machiato32 could you test it?
Set db_host to mariadb in .env. I switched to mariadb to avoid possible conflicts.

@machiato32


v1.4 bug fix

Codecov Report

Merging #241 (cc468a8) into master (006e85d) will decrease coverage by 0.18%.
The diff coverage is 45.16%.

@@             Coverage Diff              @@
##             master     #241      +/-   ##
============================================
- Coverage     63.12%   62.93%   -0.19%
+ Complexity      831      830       -1
============================================
  Files           138      138
  Lines          3159     3157       -2
  Branches        410      409       -1
============================================
- Hits           1994     1987       -7
- Misses         1023     1026       +3
- Partials        142      144       +2

Impacted Files
Coverage Δ

...du/address/logic/parser/EditTaskCommandParser.java
70.00% <0.00%> (-7.78%)
:arrow_down:

.../main/java/seedu/address/model/task/EventTask.java
15.62% <0.00%> (+1.33%)
:arrow_up:

src/main/java/seedu/address/model/task/Task.java
37.33% <0.00%> (-10.77%)
:arrow_down:

...c/main/java/seedu/address/model/task/TaskDate.java
55.55% <ø> (-1.59%)
:arrow_down:

.../seedu/address/logic/commands/EditTaskCommand.java
71.59% <50.00%> (-1.91%)
:arrow_down:

...edu/address/logic/commands/AddTodoTaskCommand.java
100.00% <100.00%> (ø)

...address/logic/parser/AddTodoTaskCommandParser.java
80.00% <100.00%> (ø)

...in/java/seedu/address/model/task/DeadlineTask.java
75.00% <100.00%> (+9.28%)
:arrow_up:

...c/main/java/seedu/address/model/task/TodoTask.java
75.00% <100.00%> (+1.66%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 006e85d...cc468a8. Read the comment docs.


Add ability to load access token from environment variable

It looks like it's only possible to specify an access token using -A or with a config file.
Would it be possible to allow this to be set using an environment variable?


Finished step 4 of week 3 walkthrough

Add form and doPost method along with creating box for easier visual for comments section. @yjennywang please look at this when you get a chance! Thanks:)
here's a picture of the comments box so you can see how I changed it with CSS.

@yjennywang please take a look when you can!


[BUG] Reload search analyzer not working on cluster

Describe the bug
Hello everyone! I tried using this API operation, but its not working on cluster. But it gives:
I tried this on 2.7.0 and got the following error:
{
    "error": "no handler found for uri [/books/_reload_search_analyzers] and method [POST]"
}

To Reproduce
Run cluster
Create index
Use URI from documentation "GET /{index}/_reload_search_analyzers" or "POST /{index}/_reload_search_analyzers"
Expected behavior
Response like from documentation:
{
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "reload_details": [
    {
      "index": "shakespeare",
      "reloaded_analyzers": [
        "analyzers-synonyms-test"
      ],
      "reloaded_node_ids": [
        "opensearch-node1"
      ]
    }
  ]
}

Host/Environment (please complete the following information):
You can run from docker or use OpenSearch playground, that doesn't matter.
Additional context
Add any other context about the problem here.

I can't find any _reload_search_analyzers in server code, only clients. Which version of ES was this added in?

@nhtruong I'm just trying to understand what happened to the feature first. If it was added to ES later than 7.10 we should remove it from OpenSearch docs, yes.

Taking a look at the Elastic documentation, reload search analyzers were not available for general availability in 7.10, but were available in 7.11. We should remove it from OpenSearch docs.

Great, thanks. We also need to remove it from clients. I'll transfer this to the documentation repo to start and cc: @wbeckler for clients visibility.


Submit Smoldyn 2.72

id: smoldyn
version: 2.72
specificationsUrl: https://raw.githubusercontent.com/ssandrews/Smoldyn/master/biosimulators.json
specificationsPatch:
version: 2.72
image:
url: ghcr.io/ssandrews/smoldyn/biosimulators_smoldyn:2.72
digest: sha256:96d9ed99272eec7bc72c310f72de401043fcc8869ebd9c7482c96495d11764c4
validateImage: true
commitSimulator: true

Thank you @AlexPatrie for your submission to the BioSimulators simulator validation/submission system!
The BioSimulators validator bot is validating the specifications of your simulator, validating your Docker image, and committing your simulator to the BioSimulators registry.
We will discuss any concerns with your submission in this issue.
A complete log of your simulator submission job will be available for 90 days here. The results of the validation of your tool will also be saved as a JSON file. A link to this file will be available for 90 days from the "Artifacts" section at the bottom of this page.

The validation/submission of your simulator failed.
- The simulator specifications from `https://raw.githubusercontent.com/ssandrews/Smoldyn/master/biosimulators.json` are invalid. The specifications of simulation tools must adhere to BioSimulators' schema. BioSimulators' schema is available in both JSON Schema and Open API Specifications formats. Documentation is available at https://api.biosimulators.org/.
-
-   - Bad Request (400):
-
-       Object is invalid: Property or parameter 'image' is invalid.
-
-         Value:
-           Image {
-             operatingSystemType: 'Linux',
-             url: 'ghcr.io/ssandrews/smoldyn/biosimulators_smoldyn:2.72',
-             format: EdamOntologyDockerImageIdVersion {
-               version: null,
-               supportedFeatures: [],
-               namespace: 'EDAM',
-               id: 'format_3973'
-             }
-           }
-
-         Error(s):
-           - Property or parameter 'digest' is invalid.
-
-               Value:
-                 undefined
-
-               Error(s):
-                 - isImageDigest: An image digest must be a valid sha256 hash with the "sha256:" prefix
-                 - isNotEmpty: digest should not be empty

The complete log of your validation/submission job, including further information about the failure, is available for 90 days here.
If you chose to validate your Docker image, the results of the validation of your image will be available shortly as a JSON file. A link to this file will be available for 90 days from the "Artifacts" section at the bottom of this page.
Once you have fixed the problem, edit the first block of this issue to re-initiate this validation.
The BioSimulators Team is happy to help. Questions and feedback can be directed to the BioSimulators Team by posting comments to this issues that reference the GitHub team @biosimulators/biosimulators (without the backticks).

Thank you @AlexPatrie for your submission to the BioSimulators simulator validation/submission system!
The BioSimulators validator bot is validating the specifications of your simulator, validating your Docker image, and committing your simulator to the BioSimulators registry.
We will discuss any concerns with your submission in this issue.
A complete log of your simulator submission job will be available for 90 days here. The results of the validation of your tool will also be saved as a JSON file. A link to this file will be available for 90 days from the "Artifacts" section at the bottom of this page.

- You (@AlexPatrie) do not have permissions to update the specifications of Smoldyn. Only the members of @biosimulators/smoldyn can update the specifications of Smoldyn. Please contact the members of this group to request permissions to update Smoldyn.

Thank you @AlexPatrie for your submission to the BioSimulators simulator validation/submission system!
The BioSimulators validator bot is validating the specifications of your simulator, validating your Docker image, and committing your simulator to the BioSimulators registry.
We will discuss any concerns with your submission in this issue.
A complete log of your simulator submission job will be available for 90 days here. The results of the validation of your tool will also be saved as a JSON file. A link to this file will be available for 90 days from the "Artifacts" section at the bottom of this page.

- You (@AlexPatrie) do not have permissions to update the specifications of Smoldyn. Only the members of @biosimulators/smoldyn can update the specifications of Smoldyn. Please contact the members of this group to request permissions to update Smoldyn.


User confirmation not properly processed when run as a container

When running tcpgoon in a container, the user confirmation is not properly processed:
****************************** WARNING ******************************
* You are going to run a TCP stress check with these arguments:
*	- Host: www.google.com
*	- TCP Port: 80
*	- # of concurrent connections: 5
*********************************************************************
Do you want to continue? (y/N): Response not processed```

layer8 issue


Flatten the testing matrix

We're running into concurrency limits on macOS in the CI and it's also just wasting compute. We should flatten the testing matrix and run the various tests suites serially instead of in parallel on separate machines. This will save the cost of repeatedly doing all the npm setup.

I tried to do so but seems like it also takes quite a lot of time. Most of the time spent is on running the e2e test and not on doing npm setup since we already have cached deps. You can take a look at #1858 in case that is what you mean in this issue

Thanks for the time you put into this - I think, rather than overall time, what we were going for here was fewer separate machines used to keep us away from the concurrency limits. If it's going to take a lot longe to execute them, we might want to just keep doing it the way we're doing it. As you said, everything is cached, so the setup overhead is relatively low.
Let's just abandon this for now. Thanks again for the effort!


Add support for DSIG digital signature in OpenType and TrueType (Collection) fonts

Strange that there is no issue so far. So digital signature is the only way to be sure that the font is not a "rip" and has all kerning/hinting/ligatures/glyphs in tact... It is a problem because the only implementation is from Microsoft (that actually verifies the signature, because some clever people just insert NULL DSIG, hehe) and in 32 bit library only, the 64 bit mssipotf.dll library is even apparently prohibited to destribute (see discussion https://github.com/fonttools/fonttools/issues/306).
Of course I can also remind you about CVEs by using malformed font files. 32 bit dll is here. https://docs.microsoft.com/en-us/typography/tools/signing-tool/

DSIG seems to be specified here:
https://docs.microsoft.com/en-us/typography/opentype/spec/dsig
And the actual signature itself inside it is based on PKCS7 (CMS):

Create the PKCS#7 signature block using the content digest.

OpenSSL already has CMS support, so it should already be possible to do this.

I will also point out for your comfort this mailing list email that poinyed out the only mistake of the spec. https://docs.microsoft.com/en-us/typography/tools/signing-tool/.  https://lists.aau.at/pipermail/mpeg-otspec/2016-December/000837.html

a) update the spec to match the implementation (i.e. remove the line "Zero
out the file checksum in the head table.", as it is not actually
performed by signcode.exe + mssipotf.dll, as I and Hin-Tak have repeatedly
explained);

Here are some extra description, nice! https://github.com/sladen/fontsign in particular it uses OID <IP_ADDRESS>.4.1.311.40.1
I personally used Cerbero Profiler to see binary but whatever.

The PKCS#7 block in the dsig table is an ASN1 structure that contains an Authenticode-style signature. This is only tangentially related to OpenSSL. OpenSSL is really only needed to sign the content digest of the font file. The rest can be built using some lib that can create ASN1 structures.

tangentially related to OpenSSL

Sure. Except it was never implemented in open source. Which is so not COOL! So... BTW I extracted the DSIG table in fool, here is a link  to ASN.1 parser

It is not true that no other signing implementation exists other than Microsoft's. There is one based on openssl, developed in-house at Dalton Maag, and perhaps another with the fontlab people too. I had the former's prototype code in a private communication with Dalton Maag folks a few years ago.

And I opened https://github.com/microsoft/Font-Validator/issues/44 a few years ago...

I don't think the 64-bit dll is "prohibited from distribution/redistribution" - but since it is Microsoft's, it is up to them to distribute it. If they haven't posted them publicly for whatever reasons, and one of them chose to send it to some, you have got to respect their intention: if it is not posted in public, it is probably not supposed to.

Dalton Maag

When that company, whatever it is, never heard of it, will submit pull request with sign off here I will believe you. Also it is not a real implementation, one can extract that table without much problems and to add OID is not a problem.

it is not posted in public, it is probably not supposed to.

So it IS prohibited for destribution. Who evn uses 32 bit code in 2020? Lol.

Dalton Maag made the Ubuntu system font for Ubuntu. Look at the copyright notice of the Ubuntu font.
The official 32-bit signing tool apparently still works quite well on wow64.
Anyway, I prefer not to respond to this thread anymore.

system font for Ubuntu

And Debian made Ubuntu. So what? Also I use MacOS fonts on MY DEBIAN.

Search for "Dalton Maag" in this page https://design.ubuntu.com/font .

Windows XP source code leaked. We now have all mssipotf.dll source code. The spec will be also corrected soon.


ARROW-7181: [C++] Fix an Arrow module search bug with pkg-config

If an Arrow module has only shared library and pkg-config is used to
find the module, FindArrow.cmake can't find directory that has the
shared library.

@ursabot crossbow submit -g conda -g wheel

LGTM, wheels are passing. conda-osx is a different issue, ticketed at https://issues.apache.org/jira/browse/ARROW-7189.


Issue with findModeByExtension function in mode/meta.js

In mode/meta.js file, the findModeByExtension function doesn't return anything if a match wasn't found.
e.x. if you try the code:
editor.setOption("mode", CodeMirror.findModeByFileName(this.querySelector('.filename').innerHTML).mode);
will work on every supported filename but on unsupported ones will return undefined
This issue can be resolved easily by placing return("null"); as the last line of the findModeByExtension function.

JavaScript functions will return undefined if they fall off the bottom of the function body, so there's no need to add an explicit return statement when undefined is an acceptable return value.


Update the county_limit_data_latest.csv file for 2021

This is the annual prep step for updating county mortgage limits
for the owning-a-home API. In addition to updating source files
for 2021, this PR also cleans up old files and folders that we can relegate
to git history.
After the first of the year, we can run the Jenkins cf.gov-load-oah-county-limits
job to update the production db.
Testing
The new limit data have been loaded on DEV2, so you can compare API calls there against production and see that the limit values have gone up. Example (Ignore the stray giant icon; it's unrelated to this PR):
prod: https://www.consumerfinance.gov/oah-api/county/?state=FL
dev2: https://dev2.demo.cfpb.gov/oah-api/county/?state=FL

merging this data update for the new year.


Nplus insights criticality

Closed as a duplicate of https://github.com/digma-ai/digma-ui/issues/316


IDEA Ciphersuites in TLS 1.2

During our research on TLS-Attacker we discovered that Openssl 1.0.2o allows the IDEA ciphersuite to be negotiated for TLS 1.2. The client accepts an IDEA cipher suite selection from a server and the server is also willing to select this cipher suite themselfs. The RFC 5246 deprecated the IDEA cipher suite:
Removed IDEA and DES cipher suites.  They are now deprecated and
will be documented in a separate document

We do not think that this is a direct security issue, but this additionally creates an interoperability issue with potential other clients since such handshakes probably result in a failed handshake.
Best regards
Robert Merget (@ic0ns), Simon Friedberger (@Apromixately) and Juraj Somorovsky (@jurajsomorovsky)

As of OpenSSL 1.1.0, IDEA, DES, and several other things are not available at the default security level.
Making this change in the 1.0.2 series would be a compatibility-breaking behavior change and seems unlikely to be done in the absence of an exploitable critical flaw.  As such, I don't think there's anything to do other than close this issue (since it is already addressed in the latest releases), but feel free to continue the conversation if you disagree with this assessment.


How can i get the visitor name to show in the Agent chat?

How can i send the visitor name to Agent chat?
I want to know with which visitor I'm talking about..

Add an input in the chat box widget and on new chat request pass that value to the ChatHub method. Have the input to hide after their first message something like that.


Update version docs

I don't think we need an example here (don't want to hard code a specific version that will either get out of date or that we will need to update with each release), and updated the docs to clarify that download will only happen if necessary.

@emcfarlane not sure why the status comment here is reporting format as failed and lint as skipped

The status comment is an integration test for this CI. It's only present on PRs to the buf-action with the fail and skip checking we report the correct values for a failed or skipped test.


Avatar is running on water instead of floating

Avatar is running on water instead of floating. There must be some splash of water or water waves should be produced.
https://user-images.githubusercontent.com/104460803/169859547-ea2260b7-2293-45a3-9358-469193ba3e7b.mp4

This is not water, Its sky, Can you try to reproduce it again? the avatar would fall when not in the limits of environment.


Unused/Unmaintained?

It looks like this project is somewhat unmaintained and no longer used by Ember CLI itself (see #28). If that is indeed the case should we put a note about that in the README file?
/cc @rwjblue @stefanpenner

that is indeed the case should we put a note about that in the README file?

👍


Mldb 1692 thread pool test

We put an environnement variable on jenkins to disable those tests.

+1
@nicolaskruchten this needs an environment variable to be set in Jenkins

@nicolaskruchten Its already setup.


Comm and battery LEDs do not turn on

I am facing problems while performing  localization and Navigation. So I tried to use Discovery server after reading similar issues (https://github.com/turtlebot/turtlebot4/issues/246#issuecomment-1691845610).
But it also didn't work for me. So I tried to switch back to simple discovery and now comm and batt LED do not turn on. Also I cannot control my robot using controller. On the robot display i can move up and down to toggle through the options but I cannot choose any of them.
For switching back to simple discovery  I changed Discovery server in 'bash setup' in turtlebot4-setup to Enabled [false].

Can you please clarify:
OS: (e.g. Ubuntu 20.04)
ROS Distro: (e.g. Galactic)
Built from source or installed:
Package version: (if from repository, give version from sudo dpkg -s ros-$ROS_VERSION-turtlebot4-PACKAGE_WITH_ISSUE, if from source, give commit hash)
It doesn't sound like you reverted from Discovery Server properly, but the details on how to fix it will depend on what Distro that you are running. If you provide more details about the core issues that led you to try discovery server then we can also discuss if troubleshooting the discovery server setup is worthwhile vs reverting to simple discovery.

I am working with
os: ubuntu 22.04.03
ROS2: Humble
Installed: "sudo apt install ros-humble-turtlebot4-desktop"
I am working on navigation using an external map that I have created using slam. I get these errors in the logs.
In localisation: frame 'rplidar_link' at time<PHONE_NUMBER>.998 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
[rviz2-1] Warning: Invalid frame ID "map" passed to canTransform argument target_frame - frame does not exist
when I launch localisation and navigation launch files I need to wait for 5-10 minutes for tf tree to show map and odom as well.
and cost map also takes this much time to load.
And after I give a nav goal point the robot turns and then nothing happens and the log is like this in navigation.
[bt_navigator-5] [INFO] [1692872679.972808500] [bt_navigator]: Begin navigating from current location (0.59, 0.12) to (0.07, 0.86)
[bt_navigator-5] [WARN] [1692872680.013463789] [bt_navigator_navigate_to_pose_rclcpp_node]: Timed out while waiting for action server to acknowledge goal request for follow_path
[bt_navigator-5] [WARN] [1692872680.033798913] [bt_navigator_navigate_to_pose_rclcpp_node]: Node timed out while executing service call to local_costmap/clear_entirely_local_costmap.
[bt_navigator-5] [WARN] [1692872680.054291683] [bt_navigator_navigate_to_pose_rclcpp_node]: Node timed out while executing service call to local_costmap/clear_entirely_local_costmap.
[behavior_server-4] [INFO] [1692872680.054674974] [behavior_server]: Running spin
[behavior_server-4] [INFO] [1692872680.054769234] [behavior_server]: Turning 1.57 for spin behavior.
[ERROR] [controller_server-1]: process has died [pid 16391, exit code -11, cmd '/opt/ros/humble/lib/nav2_controller/controller_server --ros-args --log-level info --ros-args -r __ns:=/ --params-file /tmp/tmpwizu8_td -r /tf:=tf -r /tf_static:=tf_static -r cmd_vel:=cmd_vel_nav'].
[behavior_server-4] [INFO] [1692872682.555000572] [behavior_server]: spin completed successfully
[ERROR] [bt_navigator-5]: process has died [pid 16399, exit code -11, cmd '/opt/ros/humble/lib/nav2_bt_navigator/bt_navigator --ros-args --log-level info --ros-args -r __node:=bt_navigator -r __ns:=/ --params-file /tmp/tmpoidr2378 -r /tf:=tf -r /tf_static:=tf_static'].
[lifecycle_manager-8] [INFO] [1692872684.105955493] [lifecycle_manager_navigation]: Have not received a heartbeat from controller_server.
[lifecycle_manager-8] [ERROR] [1692872684.105996957] [lifecycle_manager_navigation]: CRITICAL FAILURE: SERVER controller_server IS DOWN after not receiving a heartbeat for 4000 ms. Shutting down related nodes.
[lifecycle_manager-8] [INFO] [1692872684.106006041] [lifecycle_manager_navigation]: Terminating bond timer...
[lifecycle_manager-8] [INFO] [1692872684.106012467] [lifecycle_manager_navigation]: Resetting managed nodes...
[lifecycle_manager-8] [INFO] [1692872684.106018512] [lifecycle_manager_navigation]: Deactivating velocity_smoother
[velocity_smoother-7] [INFO] [1692872684.106718762] [velocity_smoother]: Deactivating
[velocity_smoother-7] [INFO] [1692872684.106772478] [velocity_smoother]: Destroying bond (velocity_smoother) to lifecycle manager.
[lifecycle_manager-8] [INFO] [1692872684.212621190] [lifecycle_manager_navigation]: Deactivating waypoint_follower
[waypoint_follower-6] [INFO] [1692872684.212900307] [waypoint_follower]: Deactivating
[waypoint_follower-6] [INFO] [1692872684.212947148] [waypoint_follower]: Destroying bond (waypoint_follower) to lifecycle manager.
[lifecycle_manager-8] [INFO] [1692872684.320141704] [lifecycle_manager_navigation]: Deactivating bt_navigator

@Saurabh0325 It does sound like it may be advantageous for you to try this with Discovery server. Are you on board with trying to troubleshoot discovery server? If so can you provide the following:

If you ssh into the turtlebot, run cat /etc/turtlebot4/setup.bash and share the output.

If you go to the create3 webserver, can you share a screenshot like the following:

On your PC, can you tell me what modifications you have made to your ~/.bashrc for Turtlebot and what those files that are included contain.

Thank you for your reply.
I want to drive multiple robots after I am able to achieve navigation using only one robot. I went through the user manual, there I found out that controlling multiple robots using discovery server is not possible. Anyhow I was able to revert back to simple discovery successfully.
But the navigation issue is still there. (nav2 deactivates after setting the pose and giving a goal pose). As I have mentioned above. Can you please help me with that?

@Saurabh0325 With your system set up for simple discovery:

Can you please replicate the situation and then download and share the create3 log? Please ensure to give plenty of time after the create3 boots up before starting the navigation so that it is clearly spaced out in the log.

Can you please share the exact commands that you are running to launch each piece?

Can you please still run cat /etc/turtlebot4/setup.bash on the robot and share the output?

@hilary-luo Thank you for your reply.

create3 logs: messages.txt

Localisation:   ros2 launch turtlebot4_navigation localization.launch.py map:=map_3.yaml
Nav2:   ros2 launch turtlebot4_navigation nav2.launch.py
rviz:      ros2 launch turtlebot4_viz view_robot.launch.py

Please refer logs after time stamp 14.45 for navigation related info
messages.txt

localisations: ros2 launch turtlebot4_navigation localization.launch.py map:=hiwi_raum.yaml
navigation: ros2 launch turtlebot4_navigation nav2.launch.py
rviz: ros2 launch turtlebot4_viz view_robot.launch.py

So it looks like you are running into the issue where the CPU maxes out on the create3. Can you try turning diagnostics off on the raspberry pi? (Can be done in the turtlebot4-setup tool). Let me know if that helps and if it doesn't, please provide the create3 log again of the new attempt.

Thank you for your help. I was able to perform navigation. There was an error with Fastdds package and after reinstallation and proper sourcing navigation started working.


Started working on the aspiration screen.

Added an aspiration screen

That's not how it works.

That is how it works


Wrong buy sell display data

it will write in my open order as

Basically every order made at 3 satoshi, will be written with 2 satoshi instead of 3
1-2-4-5-6 works fine, only if i enter a value of 3.
Any ideas?

Also noticed this happens randomly too @3s3s
i think its the math somewhere around here:
https://github.com/3s3s/opentrade/blob/c403833f8b50e5eba6bc3d138a090b615d04b3ae/server/utils.js#L52
if you look at this code example the output should be 0.00000003 but its outputting 0.00000002
https://www.w3schools.com/code/tryit.asp?filename=FWPAIHGL5DLB
and if you change the 0.00000003  to say 0.00000004 it outputs 0.00000004 correctly its only certain low numbers it outputs incorrectly
what are your thoughts @3s3s ?

By default, OpenTrade is rounding float values to the seventh decimal place. You can change thishttps://github.com/3s3s/opentrade/blob/c403833f8b50e5eba6bc3d138a090b615d04b3ae/server/utils.js#L47
on
decimals = 9;

Thanks!!!!! @3s3s
By change decimals from 8 to 9, orders prices now are going thru correctly.
The only issue that occoured now, after Decimals change, is that every order either from buy/sell placed at 1 satoshi, after fer seconds automatically disappear (like if there is a timeout somewhere).
If i'm quick by make a fast buy/sell at 1 satoshi, orders are recorded correctly, but if i leave orders, they disappear.
That's happened after Decimals change to 9 instead at 8 and only for 1 satoshi orders.
But i'm not sure this is related at 100% to this change on decimals.

Try to change this constant
https://github.com/3s3s/opentrade/blob/master/server/constants.js#L23
on
0.000000001

@3s3s
Thank you
It works like a charm now