Support span#error in toolkit

Please answer these questions before submitting pull request

Why submit this pull request?

[ ] Bug fix

[ ] New feature provided

[ ] Improve performance

Related issues

Bug fix

Bug description.

How to fix?

New feature or improvement

Describe the details and related test reports.

@ascrutae Do we need test case for this?

Support #2599

@IanCao  I suggest that to add error message or an exception in ActiveSpan#error  API, Because I'm confused why this span occurs an exception when the span without any error message or an exception.

I suggest that to add error message or an exception in ActiveSpan#error API, Because the user will confused why this span occurs an exception when the span without any error message or an exception.

@ascrutae Exception is a log API, do you suggest to add that?

Coverage decreased (-0.007%) to 16.411% when pulling 5220451f9c1cc5c122a1f1aac41a530b9fb2774c on IanCao:master into e6ebf62a1fa8a508fe88e17a1339be53cb47f94d on apache:master.

I suggest that to add error message or an exception in ActiveSpan#error API, Because the user will confused why this span occurs an exception when the span without any error message or an exception.

@ascrutae Exception is a log API, do you suggest to add that?

i have added three error methods and test case
   public static void error() {
    }
    public static void error(String key, String errorMsg) {
    }
    public static void error(Throwable throwable) {
    }

plz review~

@wu-sheng please review

Application toolkit document should update, and add an integration test case for this?

Application toolkit document should update, and add an integration test case for this?

ok

Application toolkit document should update, and add an integration test case for this?

doc and integration test case has added


Provide built-in fetch globally

One inconsistency I've noticed is that Next.js will provide a global fetch function in Node environments. Right now next-plugin-preval does not do that and it should!

FYI here is how I fixed this in Next Right Now, I had a similar issue in Jest tests:
/**
 * Importing next during test applies automated polyfills:
 *  - Polyfill the built-in "fetch" provided by Next.js
 *
 * @see https://github.com/vercel/next.js/discussions/13678#discussioncomment-22383 How to use built-in fetch in tests?
 * @see https://nextjs.org/blog/next-9-4#improved-built-in-fetch-support Next.js Blog - Improved Built-in Fetch Support
 * @see https://jestjs.io/docs/en/configuration#setupfilesafterenv-array About setupFilesAfterEnv usage
 */
require('next');

See https://github.com/UnlyEd/next-right-now/blob/30d83e961b6267ee7704d21b70004ee167f95ff5/jest.setup.js#L7-L15

Awesome, I wasn't sure it would be that easy! Glad it is ;)


[Backport] [2.x] Fix a bug on handling an invalid array value for point type field (#4900)

With this commit, appropriate exception is thrown when an array of four values are provided for point type field.
Signed-off-by: Heemin Kim<EMAIL_ADDRESS>(cherry picked from commit 0f477a2d0aae4d75313a7f26407cda8594bd981e)

Description
Backport of https://github.com/opensearch-project/OpenSearch/pull/4900
Issues Resolved
See https://github.com/opensearch-project/OpenSearch/pull/4900
Check List

[X] New functionality includes testing.

[X] All tests pass

[X] New functionality has been documented.

[X] New functionality has javadoc added

[X] Commits are signed per the DCO using --signoff
[X] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Test Result](https://build.ci.opensearch.org/job/gradle-check/5576/testReport/) (1 failure / +1)
[org.opensearch.search.PitMultiNodeTests.testCreatePitWhileNodeDropWithAllowPartialCreationFalse](https://build.ci.opensearch.org/job/gradle-check/5576/testReport/junit/org.opensearch.search/PitMultiNodeTests/testCreatePitWhileNodeDropWithAllowPartialCreationFalse/)

https://github.com/opensearch-project/OpenSearch/issues/4259


[DropDownMenu] Added properties menuAutoWidth and menuListStyle in order to have 100% width

I added to DropDownMenu some properties overriding Menu default properties.

One of the roadmap objectives is to standardise prop names across components where possible. Including some or all of the component name in the prop (menu) is both redundant, and counter to this objective.

What I want to achive is that both label and menu list of DropDownMenu have 100% width of it's container.
Without menuAutoWith and menuListStyle properties I wasn't able to do it.
With new properties my component is:
        <DropDownMenu
                style={{width: '100%'}} autoWidth={false}
                labelStyle={{paddingLeft: 0, paddingRight: 24}}
                menuStyle={{width: '100%'}}
                menuAutoWidth={false}
                menuListStyle={{display: 'block'}}
                value={value}
                onChange={this.onSelectAccount.bind(this)}>{menuItems}</DropDownMenu>

Sorry it took so long to review this, but unless I'm misunderstanding your use-case, in the current release you can set the menu to 100% of the container width with:
<DropDownMenu autoWidth={false} style={{width: '100%'}} >

I'm going to close this for now, but let me know if that doesn't work for you.


Updating Packages Crashes The Function When Executed

This repo is 3 years old. Any updates on the roadmap for full support or updated version of the code.

Unfortunate that Netlify choose to not go further with support for Rust functions. Seems like Vercel is the only viable choice for this at this time, in similar offerings.

for serverless/lambdas I'd suggest using https://www.cargo-lambda.info/ and AWS personally.


Fantastic csv error message

github gets this one right - https://github.com/crbclean/leads/blob/8ea5f134dd0d63ef6d8eec17f4757b0c7b62b2b3/Alabama2.csv

Fixed this the old fashioned way - they are using escape characters to avoid quoting things like newlines.  escape is now an optional parameter that defaults to \.


Bump to 0.1.2

Coverage remained the same at 100.0% when pulling 17b5cfcdeca6a68e70bcea7660c248e1d0170ada on ljqx:bump-to-0.1.2 into dd4677c6b61c302c95756823a07d715c7ee2075f on Microsoft:publish.


Implement OCW news view

Fixes #11
This implements a simple API which converts the RSS feed for OCW News to JSON. Not all content from the RSS feed is kept, but there should be enough information to create a widget on the ocw-www front page for https://github.com/mitodl/ocw-www/issues/6

Codecov Report

Merging #12 (910094c) into master (783ccc8) will increase coverage by 12.87%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master      #12       +/-   ##
===========================================
+ Coverage   76.48%   89.36%   +12.87%
===========================================
  Files          25        4       -21
  Lines         621       47      -574
  Branches       57        0       -57
===========================================
- Hits          475       42      -433
+ Misses        132        5      -127
+ Partials       14        0       -14

Impacted Files
Coverage Δ

main/templatetags/render_bundle.py

main/management/commands/generate_app_json.py

main/admin.py

main/apps.py

main/sentry.py

users/models.py

websites/constants.py

main/__init__.py

main/celery.py

main/envs.py

... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 783ccc8...910094c. Read the comment docs.

Codecov Report

Merging #12 (910094c) into master (783ccc8) will increase coverage by 12.87%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master      #12       +/-   ##
===========================================
+ Coverage   76.48%   89.36%   +12.87%
===========================================
  Files          25        4       -21
  Lines         621       47      -574
  Branches       57        0       -57
===========================================
- Hits          475       42      -433
+ Misses        132        5      -127
+ Partials       14        0       -14

Impacted Files
Coverage Δ

main/templatetags/render_bundle.py

main/management/commands/generate_app_json.py

main/admin.py

main/apps.py

main/sentry.py

users/models.py

websites/constants.py

main/__init__.py

main/celery.py

main/envs.py

... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 783ccc8...910094c. Read the comment docs.

@mbertrand Actually I forgot about a few changes I need to make so I'm moving this back to work in progress

@mbertrand Actually I forgot about a few changes I need to make so I'm moving this back to work in progress

@mbertrand Should be ready now

@mbertrand Should be ready now


an exception is being thrown when using a list in a post body

      final response = await client.put(
        <Route obscured for privacy>,
        body: [
          {'id': id}
        ],
      );

when i send an array/list in the body it throws an exception where List<Map<String,dynamic> not subtype of Map<String,dynamic>
when showing the request in the logs screen
httpclient used is Dio

Hey! Maybe i fixed this in #20
But is not merged yet...
You can use is like
dependency_overrides:
  cr_logger:
    git:
      url: https://github.com/DizoftTeam/cr_logger.git

Even library updated 2 weeks ago this 2 years issue still remain. I am not sure that there is big amount of work.


tests reqs: pin pytest-django==2.8.0

Ref: https://github.com/yourlabs/django-autocomplete-light/pull/501#issuecomment-145834714

@jpic
I am going ahead and merge this for now.


Death stranding dc

Should fix #35 as well as support the standard version released before the DC.

You might as well close this and just open a PR against main instead

Okay, I'll create one towards main. Was thinking in case additional changes were needed we would still have your branch
Thank you!


bug: cortex run repo_id/model_id errors "Model is not loaded yet"

Cortex version
v83
Describe the Bug
Download any non cortex built in model. e.g. cortex-nightly pull bartowski/Meta-Llama-3.1-8B-Instruct-GGUF
cortex run <model_id> doesn't work.
We get an error: "Model is not loaded yet"
Steps to Reproduce

separately cortex pull bartowski/Meta-Llama-3.1-8B-Instruct-GGUF
check the model id: /Meta-Llama-3.1-8B-Instruct-GGUF
cortex run Meta-Llama-3.1-8B-Instruct-GGUF

(base) PS C:\Users\n\cortexcpp-nightly\models> cortex-nightly --verbose run Meta-Llama-3.1-8B-Instruct-GGUF

20240923 08:25:29.333000 UTC 20564 INFO  Config file name: .cortexrc-nightly - utils/file_manager_utils.h:105
20240923 08:25:29.333000 UTC 20564 INFO  Config file name: .cortexrc-nightly - utils/file_manager_utils.h:105
20240923 08:25:29.336000 UTC 20564 INFO  Config file name: .cortexrc-nightly - utils/file_manager_utils.h:105
20240923 08:25:29.338000 UTC 20564 INFO  Config file name: .cortexrc-nightly - utils/file_manager_utils.h:105
20240923 08:25:29.338000 UTC 20564 INFO  Config file name: .cortexrc-nightly - utils/file_manager_utils.h:105
20240923 08:25:36.036000 UTC 20564 INFO  Model is not loaded yet! - chat_cmd.cc:56
20240923 08:25:36.036000 UTC 20564 INFO  Engine release path: https://delta.jan.ai/cortex/latest/version.json - commands/cortex_upd_cmd.h:69

Expected

see server started
loading model
etc.
it seemed to have skipped all of these steps

What is your OS?

[ ] MacOS
[X] Windows
[ ] Linux

What engine are you running?

[X] cortex.llamacpp (default)
[ ] cortex.tensorrt-llm (Nvidia GPUs)
[ ] cortex.onnx (NPUs, DirectML)

This issue should be resolved on cortex v0.5.0-98. Can you please try it out? @0xSage

This issue should be resolved on cortex v0.5.0-98. Can you please try it out? @0xSage

Not fixed. On version v116
❯ cortex-nightly run bartowski/Meta-Llama-3.1-8B-Instruct-GGUF
Select an option
1. Meta-Llama-3.1-8B-Instruct-IQ2_M.gguf
2. Meta-Llama-3.1-8B-Instruct-IQ3_M.gguf
3. Meta-Llama-3.1-8B-Instruct-IQ3_XS.gguf
4. Meta-Llama-3.1-8B-Instruct-IQ4_XS.gguf
5. Meta-Llama-3.1-8B-Instruct-Q2_K.gguf
6. Meta-Llama-3.1-8B-Instruct-Q2_K_L.gguf
7. Meta-Llama-3.1-8B-Instruct-Q3_K_L.gguf
8. Meta-Llama-3.1-8B-Instruct-Q3_K_M.gguf
9. Meta-Llama-3.1-8B-Instruct-Q3_K_S.gguf
10. Meta-Llama-3.1-8B-Instruct-Q3_K_XL.gguf
11. Meta-Llama-3.1-8B-Instruct-Q4_K_L.gguf
12. Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
13. Meta-Llama-3.1-8B-Instruct-Q4_K_S.gguf
14. Meta-Llama-3.1-8B-Instruct-Q5_K_L.gguf
15. Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf
16. Meta-Llama-3.1-8B-Instruct-Q5_K_S.gguf
17. Meta-Llama-3.1-8B-Instruct-Q6_K.gguf
18. Meta-Llama-3.1-8B-Instruct-Q6_K_L.gguf
19. Meta-Llama-3.1-8B-Instruct-Q8_0.gguf
20. Meta-Llama-3.1-8B-Instruct-f32.gguf

Select an option (1-20): 1
Selected: Meta-Llama-3.1-8B-Instruct-IQ2_M.gguf
Validating download items, please wait..
Start downloading: Meta-Llama-3.1-8B-Instruct-IQ2_M.gguf
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1158  100  1158    0     0   4039      0 --:--:-- --:--:-- --:--:--  4048
100 2811M  100 2811M    0     0  5892k      0  0:08:08  0:08:08 --:--:-- 6711k
Model Meta-Llama-3.1-8B-Instruct-GGUF downloaded successfully!
Fail to run model with ID 'bartowski/Meta-Llama-3.1-8B-Instruct-GGUF': Model not found: bartowski/Meta-Llama-3.1-8B-Instruct-GGUF

Running model list, saved ID & alias are different than the HF model ID, thus model doesnt load.
❯ cortex-nightly models list
+---------+------------------------------------------------------------------------------------------------+----------------------------------+-----------------+---------+
| (Index) |                                               ID                                               |            model alias           |      engine     | version |
+---------+------------------------------------------------------------------------------------------------+----------------------------------+-----------------+---------+
|    1    | tinyllama:gguf                                                                                | tinyllama:gguf                   | cortex.llamacpp |    1    |
+---------+------------------------------------------------------------------------------------------------+----------------------------------+-----------------+---------+
|    2    | huggingface.co/hugging-quants/Llama-3.2-3B-Instruct-Q8_0-GGUF/llama-3.2-3b-instruct-q8_0.gguf  | llama-3.2-3b-instruct-q8_0       | cortex.llamacpp |    2    |
+---------+------------------------------------------------------------------------------------------------+----------------------------------+-----------------+---------+
|    3    | huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/Meta-Llama-3.1-8B-Instruct-IQ2_M.gguf | meta-llama-3.1-8b-instruct-iq2_m | cortex.llamacpp |    2    |
+---------+------------------------------------------------------------------------------------------------+----------------------------------+-----------------+---------+

cortex run on v116 is broken. We just had a fix for it on v117. Apologies for the inconvenience

Hi @vansangpfiev,
I'm getting this on the ubuntu-app-cpu-1 VM
After running cortex-nightly run bartowski/Meta-Llama-3.1-8B-Instruct-GGUF successfully and model downloaded.
Not sure why Model not found - is it to do with the model ID vs Alias vs author/model_id?

On Mac v140
similarly: ran cortex-nightly run bartowski/Meta-Llama-3.1-8B-Instruct-GGUF successfully and model downloaded.
returns
Error: Please delete the model before downloading again```

![Image](https://github.com/user-attachments/assets/89e6c77e-dcf6-4f79-912e-6dbbe7ce429d)

I'm going to close this issue as "Model is not loaded yet" error has been resolved
Tracking the above followup "Please delete the model before downloading again" in #1408


IO features for V1

Looks like there are some conflicts with master here

Thanks for putting this in @BradReesWork!


orc: 0.4.38 -> 0.4.39

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/project/orc/versions.
meta.description for orc is: Oil Runtime Compiler
meta.homepage for orc is: https://gstreamer.freedesktop.org/projects/orc.html
meta.changelog for orc is: https://cgit.freedesktop.org/gstreamer/orc/plain/RELEASE?h=0.4.39
Updates performed

Version update

To inspect upstream changes
Impact
Checks done

built on NixOS
Warning: a test defined in passthru.tests did not pass
found 0.4.39 with grep in /nix/store/6r5slv82pvn0m4dz62qg745741j6jpyc-orc-0.4.39
found 0.4.39 in filename of file in /nix/store/6r5slv82pvn0m4dz62qg745741j6jpyc-orc-0.4.39

Rebuild report (if merged into master) (click to expand)

4316 total rebuild path(s)

4316 package rebuild(s)

First fifty rebuilds by attrpath

CuboCore.corepdf
CuboCore.coretime
CuboCore.coretoppings
MMA
OVMFFull
SDL2
SDL2_Pango
SDL2_gfx
SDL2_image
SDL2_image_2_0
SDL2_image_2_6
SDL2_mixer
SDL2_mixer_2_0
SDL2_net
SDL2_sound
SDL2_ttf
SDL_audiolib
SDL_compat
SDL_gpu
_1oom
_20kly
_2ship2harkinian
_64gram
_7kaa
_86Box
_86Box-with-roms
_90secondportraits
aaxtomp3
abbaye-des-morts
abracadabra
abuse
actiona
activitywatch
adapta-gtk-theme
adl
adriconf
adwsteamgtk
aegisub
affine
ags
ahoviewer
aitrack
ajour
akkoma
alarm-clock-applet
alass
alephone
alfaview
alfis

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/6r5slv82pvn0m4dz62qg745741j6jpyc-orc-0.4.39 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A orc https://github.com/r-ryantm/nixpkgs/archive/adcf839de2f2fd310211b49c9a896fc06a8a8404.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/adcf839de2f2fd310211b49c9a896fc06a8a8404#orc

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/6r5slv82pvn0m4dz62qg745741j6jpyc-orc-0.4.39
ls -la /nix/store/6r5slv82pvn0m4dz62qg745741j6jpyc-orc-0.4.39/bin

Pre-merge build results
NixPkgs review skipped

Maintainer pings

0.4.39
======

- Security: Fix error message printing buffer overflow leading to possible
  code executation in orcc with specific input files (CVE-2024-40897). This
  only affects developers and CI environments using orcc, not users of liborc
  (Sebastian Dröge, L. E. Segovia)
- div255w: fix off-by-one error in the implementations (L. E. Segovia)
- x86: only run AVX detection if xgetbv is available (L. E. Segovia)
- x86: fix AVX detection by implementing the check recommended by Intel (L. E. Segovia)
- Only enable JIT compilation on Apple arm64 if running on macOS, fixes crashes on iOS (Piotr Brzeziński)
- Fix potential crash in emulation mode if logging is enabled (L. E. Segovia)
- Handle undefined TARGET_OS_OSX correctly (Joshua Root)
- orconce: Fix typo in GCC __sync-based implementation (Jakub Vaněk)
- orconce: Fix usage of __STDC_NO_ATOMICS__ (Alexander Slobodeniuk)
- Fix build with MSVC 17.10 + C11 (L. E. Segovia)
- Support stack unwinding on Windows (L. E. Segovia)
- Major opcode and instruction set code clean-ups and refactoring (Jorge Zapata)
- Refactor allocation and chunk initialization of code regions (L. E. Segovia)
- Fall back to emulation on Linux if JIT support is not available,
  e.g. because of SELinux sandboxing or noexec mounting) (L. E. Segovia)


uniq() table function

It would be useful to have a function similar to joining() , but a list of unique values is returned.
Happy to accept the usual caveats that joining() presents.

Going to call it distinct() as that is more sql than linux shell


Add support for OpenRC runlevels

Alpine Linux (and Gentoo) use OpenRC. Goss was hard-coded to expect a service to be at the sysinit runlevel, if the service was at a different runlevel the service check would erroneously fail.
Behaviour prior to this PR: -
alpine38:/tmp# service sshd status
 * status: started

alpine38:/tmp# ./goss-0.3.6-linux-amd64 -g /tmp/goss/goss.yaml v
..........F.

Failures/Skipped:

Service: sshd: enabled:
Expected
    <bool>: false
to equal
    <bool>: true

Total Duration: 0.012s
Count: 12, Failed: 1, Skipped: 0

With this PR the service block now supports an optional runlevel attribute, which defaults to sysinit if omitted maintaining the old behaviour.
Now it is possible to install openssh-server in Alpine linux and properly check the service is running with the following config: -
service:
  sshd:
    enabled: true
    running: true
    runlevel: default

Additionally you might have:
port:
  tcp:22:
    listening: true
    ip:
    - <IP_ADDRESS>
  tcp6:22:
    listening: true
    ip:
    - '::'
user:
  sshd:
    exists: true
    groups:
    - sshd
    home: /dev/null
    shell: /sbin/nologin
group:
  sshd:
    exists: true
process:
  sshd:
    running: true

Post this PR:
alpine38:/tmp# ~/goss-berne2d -g /tmp/goss/goss.yaml v
runlevel is default
............

Total Duration: 0.012s
Count: 12, Failed: 0, Skipped: 0

Note: I'm quite new to reading and writing golang. I've tested the changes and the code works for me. If I have used bad constructs or poor programming please correct it or help me correct it.
Ideally I'd have written test cases but I don't know how to do that yet.

Hmm on my system its pulling in the newer version of mount so it wants a filter (I raised PR #370 for this) but in travis-ci it's using the old version and complains of too many arguments. I don't understand glide and golang enough to understand why the dependency is upgraded on my system but not in travis-ci. I'm using Ubuntu 16.04 LTS , go v1.10.3, glide v0.13.1.

I rebased onto master to remove the GetMounts change (#370). I also made the new runlevel attribute omitted if it is blank, which makes all the integration tests pass.
I fixed the other make test errors, updating the Dockerfile for Wheezy to fix apache2.2 package error and the corresponding goss files.
There is a warning about the hash in the  glide.lock file not matching the glide.yaml file. This appears to be from preexisting changes where the glide.lock file was not updated after changes to the glide.yaml file. This PR hasn't touched those files.
travis-ci test is now passing.

I've rebased to master.
I've updated the documentation to include the optional runlevel attribute and its default if omitted.
I've updated the integration tests. I modified the alpine3 Dockerfile, the MD5 sum won't match anymore. I have not updated the MD5 sum, since @aelsabbahy will need to rebuild the docker images.

@aelsabbahy Can this PR be re-opened? I've updated my branch to not use the struct tag default and reflection, to be consistent with the rest of goss.

Github won't let me click the  "reopen and comment" because "the *** branch was forced-pushed or recreated"
Can you create a new PR and reference it here in a comment.
Sorry for the inconvenience

Github won't let me click the "reopen and comment" because "the *** branch was forced-pushed or recreated"
Can you create a new PR and reference it here in a comment.
Sorry for the inconvenience

Ah sorry about that, after rebasing I did a force-push. I didn't know about this interaction with PRs.
I've opened a new PR https://github.com/aelsabbahy/goss/pull/650 to continue this.
Thanks


python.pkgs.pythonix: 0.1.0 -> 0.1.1

also fix the build
Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build python3.pkgs.pythonix

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: python3.pkgs.pythonix
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

This is broken on stable as well due to an nixUnstable update.

Success on x86_64-linux (full log)
Attempted: python3.pkgs.pythonix
Partial log (click to expand)
[0/1] Installing files.
Installing src/nix.so to /nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1/lib/python3.6/site-packages/nix.so
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1
shrinking /nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1/lib/python3.6/site-packages/nix.so
strip is /nix/store/4mf2xm9p32lzrim927yk92xhx35yaz62-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1/lib
patching script interpreter paths in /nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1
checking for references to /build in /nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1...
/nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1

@GrahamcOfBorg build python3.pkgs.pythonix

Failure on x86_64-darwin (full log)
Attempted: python3.pkgs.pythonix
Partial log (click to expand)
/nix/store/c31hd5m0i1hkyml99qh50p7f1p2wwyfb-nix-2.0.4-dev/include/nix/util.hh:23:9: warning: 'DT_DIR' macro redefined [-Wmacro-redefined]
#define DT_DIR 3
        ^
/nix/store/0yvai96flnxvsl8bn95nwcz1y7pd7xa5-Libsystem-osx-10.11.6/include/sys/dirent.h:126:9: note: previous definition is here
#define DT_DIR           4
        ^
3 warnings generated.
ninja: build stopped: subcommand failed.
builder for '/nix/store/d464qngb1ybqjfz1sblsm7r6q1lp5f6w-pythonix-0.1.1.drv' failed with exit code 1
error: build of '/nix/store/d464qngb1ybqjfz1sblsm7r6q1lp5f6w-pythonix-0.1.1.drv' failed

Success on x86_64-linux (full log)
Attempted: python3.pkgs.pythonix
Partial log (click to expand)
[0/1] Installing files.
Installing src/nix.so to /nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1/lib/python3.6/site-packages/nix.so
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1
shrinking /nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1/lib/python3.6/site-packages/nix.so
strip is /nix/store/4mf2xm9p32lzrim927yk92xhx35yaz62-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1/lib
patching script interpreter paths in /nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1
checking for references to /build in /nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1...
/nix/store/xikhja43yhgd1ph7gsgzfln10v086ffk-pythonix-0.1.1

@GrahamcOfBorg build python3.pkgs.pythonix

Failure on x86_64-darwin (full log)
Attempted: python3.pkgs.pythonix
Partial log (click to expand)
[1/5] Compiling C++ object 'src/src<EMAIL_ADDRESS>FAILED: src/src<EMAIL_ADDRESS>/nix/store/jpwsh1pzwnizdzqk2l266526j9ap1r5j-clang-wrapper-5.0.2/bin/c++ -Isrc/src@@nix@sha -Isrc -I../src -I/nix/store/2kjsky9h9h9klwb08f7h4vfkji3ii6ax-python3-3.6.5/include/python3.6m -I/nix/store/c31hd5m0i1hkyml99qh50p7f1p2wwyfb-nix-2.0.4-dev/include/nix -I/nix/store/cyb1vq511qrlca7lh0f7kscadr0ni55j-boehm-gc-7.6.6-dev/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -O3 -std=c++14 -std=c++17 -fvisibility=hidden  -MD -MQ 'src/src<EMAIL_ADDRESS>-MF 'src/src<EMAIL_ADDRESS>-o 'src/src<EMAIL_ADDRESS>-c ../src/nix-to-python.cc
../src/nix-to-python.cc:85:18: error: cannot pass object of non-trivial type 'std::__1::string' (aka 'basic_string<char, char_traits<char>, allocator<char> >') through variadic function; call will abort at runtime [-Wnon-pod-varargs]
                 showType(v));
                 ^
1 error generated.
ninja: build stopped: subcommand failed.
builder for '/nix/store/csg2rrxrbcw86b9wspvh1hhilrbha04h-pythonix-0.1.2.drv' failed with exit code 1
error: build of '/nix/store/csg2rrxrbcw86b9wspvh1hhilrbha04h-pythonix-0.1.2.drv' failed

Success on x86_64-linux (full log)
Attempted: python3.pkgs.pythonix
Partial log (click to expand)
[0/1] Installing files.
Installing src/nix.so to /nix/store/jdn9p7qfcd2gy8kxmgkzhckp2hikk6vj-pythonix-0.1.2/lib/python3.6/site-packages/nix.so
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/jdn9p7qfcd2gy8kxmgkzhckp2hikk6vj-pythonix-0.1.2
shrinking /nix/store/jdn9p7qfcd2gy8kxmgkzhckp2hikk6vj-pythonix-0.1.2/lib/python3.6/site-packages/nix.so
strip is /nix/store/4mf2xm9p32lzrim927yk92xhx35yaz62-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/jdn9p7qfcd2gy8kxmgkzhckp2hikk6vj-pythonix-0.1.2/lib
patching script interpreter paths in /nix/store/jdn9p7qfcd2gy8kxmgkzhckp2hikk6vj-pythonix-0.1.2
checking for references to /build in /nix/store/jdn9p7qfcd2gy8kxmgkzhckp2hikk6vj-pythonix-0.1.2...
/nix/store/jdn9p7qfcd2gy8kxmgkzhckp2hikk6vj-pythonix-0.1.2

@GrahamcOfBorg build python3.pkgs.pythonix

Success on x86_64-linux (full log)
Attempted: python3.pkgs.pythonix
Partial log (click to expand)
[0/1] Installing files.
Installing src/nix.so to /nix/store/05pqyx0r3ljvmm4m0g9bfar7gkf2aj9f-pythonix-0.1.3/lib/python3.6/site-packages/nix.so
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/05pqyx0r3ljvmm4m0g9bfar7gkf2aj9f-pythonix-0.1.3
shrinking /nix/store/05pqyx0r3ljvmm4m0g9bfar7gkf2aj9f-pythonix-0.1.3/lib/python3.6/site-packages/nix.so
strip is /nix/store/4mf2xm9p32lzrim927yk92xhx35yaz62-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/05pqyx0r3ljvmm4m0g9bfar7gkf2aj9f-pythonix-0.1.3/lib
patching script interpreter paths in /nix/store/05pqyx0r3ljvmm4m0g9bfar7gkf2aj9f-pythonix-0.1.3
checking for references to /build in /nix/store/05pqyx0r3ljvmm4m0g9bfar7gkf2aj9f-pythonix-0.1.3...
/nix/store/05pqyx0r3ljvmm4m0g9bfar7gkf2aj9f-pythonix-0.1.3

Failure on x86_64-darwin (full log)
Attempted: python3.pkgs.pythonix
Partial log (click to expand)
/nix/store/jpwsh1pzwnizdzqk2l266526j9ap1r5j-clang-wrapper-5.0.2/bin/c++ -Isrc/src@@nix@sha -Isrc -I../src -I/nix/store/2kjsky9h9h9klwb08f7h4vfkji3ii6ax-python3-3.6.5/include/python3.6m -I/nix/store/c31hd5m0i1hkyml99qh50p7f1p2wwyfb-nix-2.0.4-dev/include/nix -I/nix/store/cyb1vq511qrlca7lh0f7kscadr0ni55j-boehm-gc-7.6.6-dev/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -O3 -std=c++14 -std=c++17 -fvisibility=hidden  -MD -MQ 'src/src<EMAIL_ADDRESS>-MF 'src/src<EMAIL_ADDRESS>-o 'src/src<EMAIL_ADDRESS>-c ../src/eval.cc
../src/eval.cc:48:23: error: no member named '__cxa_exception_type' in 'std::exception_ptr'
    auto name = p ? p.__cxa_exception_type()->name() : "null";
                    ~ ^
1 error generated.
[3/5] Compiling C++ object 'src/src<EMAIL_ADDRESS>[4/5] Compiling C++ object 'src/src<EMAIL_ADDRESS>ninja: build stopped: subcommand failed.
builder for '/nix/store/52f1fk3n7ahrs6jfmdkqymy678r02dbk-pythonix-0.1.3.drv' failed with exit code 1
error: build of '/nix/store/52f1fk3n7ahrs6jfmdkqymy678r02dbk-pythonix-0.1.3.drv' failed

@GrahamcOfBorg build python3.pkgs.pythonix

Success on x86_64-linux (full log)
Attempted: python3.pkgs.pythonix
Partial log (click to expand)
[0/1] Installing files.
Installing src/nix.so to /nix/store/675yx901njbgcjhg9ywj7673mpw9g547-pythonix-0.1.4/lib/python3.6/site-packages/nix.so
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/675yx901njbgcjhg9ywj7673mpw9g547-pythonix-0.1.4
shrinking /nix/store/675yx901njbgcjhg9ywj7673mpw9g547-pythonix-0.1.4/lib/python3.6/site-packages/nix.so
strip is /nix/store/4mf2xm9p32lzrim927yk92xhx35yaz62-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/675yx901njbgcjhg9ywj7673mpw9g547-pythonix-0.1.4/lib
patching script interpreter paths in /nix/store/675yx901njbgcjhg9ywj7673mpw9g547-pythonix-0.1.4
checking for references to /build in /nix/store/675yx901njbgcjhg9ywj7673mpw9g547-pythonix-0.1.4...
/nix/store/675yx901njbgcjhg9ywj7673mpw9g547-pythonix-0.1.4

Success on x86_64-darwin (full log)
Attempted: python3.pkgs.pythonix
Partial log (click to expand)
[5/5] Linking target src/nix.so.
installing
install flags: install
[0/1] Installing files.
Installing src/nix.so to /nix/store/hlhcrzhfp4m2by3kxj1p1dxggvfys47x-pythonix-0.1.4/lib/python3.6/site-packages/nix.so
post-installation fixup
strip is /nix/store/8q05v40n0f7amgw70pk6bkb23r7hrzgn-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/hlhcrzhfp4m2by3kxj1p1dxggvfys47x-pythonix-0.1.4/lib
patching script interpreter paths in /nix/store/hlhcrzhfp4m2by3kxj1p1dxggvfys47x-pythonix-0.1.4
/nix/store/hlhcrzhfp4m2by3kxj1p1dxggvfys47x-pythonix-0.1.4

Failure on aarch64-linux (full log)
Attempted: python3.pkgs.pythonix
Partial log (click to expand)
running test tests/fetchMercurial.sh... [SKIP]
running test tests/signing.sh... [PASS]
running test tests/run.sh... [PASS]
running test tests/brotli.sh... [PASS]
running test tests/pure-eval.sh... [PASS]
running test tests/check.sh... [PASS]
running test tests/plugins.sh... [PASS]
All tests succeeded
cannot build derivation '/nix/store/hw8gyp0lv2991zfz007ybailj4ibxbdm-pythonix-0.1.3.drv': 1 dependencies couldn't be built
error: build of '/nix/store/hw8gyp0lv2991zfz007ybailj4ibxbdm-pythonix-0.1.3.drv' failed

Success on aarch64-linux (full log)
Attempted: python3.pkgs.pythonix
Partial log (click to expand)
[0/1] Installing files.
Installing src/nix.so to /nix/store/qak1sm4q2nn38mlp5zildplilqk4l01w-pythonix-0.1.1/lib/python3.6/site-packages/nix.so
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/qak1sm4q2nn38mlp5zildplilqk4l01w-pythonix-0.1.1
shrinking /nix/store/qak1sm4q2nn38mlp5zildplilqk4l01w-pythonix-0.1.1/lib/python3.6/site-packages/nix.so
strip is /nix/store/min150lkigznaayzpwvf8d4jdl1dqvx6-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/qak1sm4q2nn38mlp5zildplilqk4l01w-pythonix-0.1.1/lib
patching script interpreter paths in /nix/store/qak1sm4q2nn38mlp5zildplilqk4l01w-pythonix-0.1.1
checking for references to /build in /nix/store/qak1sm4q2nn38mlp5zildplilqk4l01w-pythonix-0.1.1...
/nix/store/qak1sm4q2nn38mlp5zildplilqk4l01w-pythonix-0.1.1

Success on aarch64-linux (full log)
Attempted: python3.pkgs.pythonix
Partial log (click to expand)
[0/1] Installing files.
Installing src/nix.so to /nix/store/vhnd3310gp26b09bwh8w2ssg4wb5f7jd-pythonix-0.1.4/lib/python3.6/site-packages/nix.so
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/vhnd3310gp26b09bwh8w2ssg4wb5f7jd-pythonix-0.1.4
shrinking /nix/store/vhnd3310gp26b09bwh8w2ssg4wb5f7jd-pythonix-0.1.4/lib/python3.6/site-packages/nix.so
strip is /nix/store/min150lkigznaayzpwvf8d4jdl1dqvx6-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/vhnd3310gp26b09bwh8w2ssg4wb5f7jd-pythonix-0.1.4/lib
patching script interpreter paths in /nix/store/vhnd3310gp26b09bwh8w2ssg4wb5f7jd-pythonix-0.1.4
checking for references to /build in /nix/store/vhnd3310gp26b09bwh8w2ssg4wb5f7jd-pythonix-0.1.4...
/nix/store/vhnd3310gp26b09bwh8w2ssg4wb5f7jd-pythonix-0.1.4

Success on aarch64-linux (full log)
Attempted: python3.pkgs.pythonix
Partial log (click to expand)
checking for references to /build in /nix/store/bmkkva7b86zn7sk84k5dmchf0jiaqh09-python2.7-setuptools-39.0.1...
building '/nix/store/xixyqrz7zpi1aj9ivsv32mqnvciakwvw-source.drv'...

trying https://github.com/Mic92/pythonix/archive/v0.1.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   122    0   122    0     0   1184      0 --:--:-- --:--:-- --:--:--  1184
100  6647    0  6647    0     0  31060      0 --:--:-- --:--:-- --:--:-- 31060
unpacking source archive /build/v0.1.1.tar.gz
/nix/store/qak1sm4q2nn38mlp5zildplilqk4l01w-pythonix-0.1.1


servicerouter.py: Only generate backends with valid hostname

Implement new command line parameter --check-hostname to only generate backends can be resolved by a dns server
Reason:
HAProxy has the same check when validating the configuration, so generated configuration with invalid hostnames would always fail reloading

I'm worried some about performance here (That could be a lot of extra getaddrinfo calls), as well as differences in what is seen between when we're doing the getaddrinfo checks vs. what haproxy sees.
I don't see how this eliminates entirely the chances of giving haproxy a config it will reject / refuse to reload since it is very possible (And if using mesos-dns somewhat likely) that when servicerouter runs and does it's checks everything has valid host names, but when haproxy runs it does the same getaddrinfo calls and doesn't find the hostnames.
Was there some particular system failure you saw with this? Should we maybe configure the backends purely with ip addresses rather than host-names? Was this just a hard to detect error?

Indeed, we had a problem in our production environment that the haproxy wrote a wrong configuration file and could not reload.
This was because a slave went down but tasks where still active in marathon and therefore a new config was written pointing to a hostname which doesn't exits anymore.
You are right that this might slow down the performance because of extra getaddrinfo calls, but it would solve at least some problems.
We are working on reducing the getaddrinfo calls.

So I think what we want to do with this then is to resolve the hostnames in servicerouter, and write the config always using IP addresses. Then we'll never have haproxy refuse to load the config script because the hostnames don't resolve. If a hostname doesn't resolve, then that particular config entry is dropped and we emit a log message (that way monitoring software can pick it up). That makes it so the hostnames are only ever resolved once, and if a hostname is resolvable when servicerouter runs but not when haproxy reloads, haproxy will still reload the configuration.
Sound reasonable?

Sounds like a good idea. We reworked the pull request this way.
You can now use host_ipv4 in the HAPROXY_BACKEND_SERVER_OPTIONS template. IP addresses for hostnames are cached and backends where IP addresses could not be resolved are ignored (and not cached)

Generally looks good.
Could the ip lookup cache only live for a single iteration of the config() function though, I know of a couple clusters where hostnames change what they point to quite a bit, and just caching over a single run of config() should work well as a caching point (All the tasks which are on the same host the hostname is only resolved once, we re-resolve each run once). With moving the caching there, we should cache failed lookups so we don't have growing failures (Hostnames start failing to resolve -> increased load through dozens/hundredes of hostname lookup failures for the tasks that were on those hosts -> more hosts / services fail)
Can you change the default template to use host_ipv4? I'd rather have people have the better operational behavior by default, rather than running into problems at some point down the line and finding out "oh, there was a flag I should have changed". I don't see an instance where that change would be particularly bad for any operator since haproxy is doing the same resolution itself.

When run with http callbacks config() is run every few seconds. That's why we moved the cache to the global context.
What do you think of a cache with TTL of a few minutes?

With #2115 perf should be considerably better so there should be a lot more headroom in the refresh interval (I hope)... Really should teach to consume server sent events from marathon for listening mode and do partial updates (Would fix a bunch of these issues)...
I think the logic around TTL is going to be complex / confusing to understand. I'm up for just leaving the cache global if it's that much of a time saver and adding some documentation of "If run in listening mode, DNS isn't re-resolved. Restart the process periodically to force re-resolution if desired".

updated documentation and default template.


Provide mechanism to export current diagrams & tiles as single JSON and reimport later

For certain live events we produce, we configure diagrams and tiles in MSAM for a large number of MediaLive channels (and their associated nodes. When the event is over, it would be nice to archive all of that work as a JSON and perhaps to be able to reimport it later, should we do a similar event with the same channels/nodes.

In addition to the mentioned request, please suggest if there is an easier way to configure live channels for events on a weekly basis. We may need to configure up to 60 events over the weekend and have the tiles/diagrams show up. Channels are not used again once the events are over. New channels implies new effort to configure the dashboard to be current and active. Please provide guidance on whether any kind of auto-discovery in MediaLive/MediaPackage is possible? Thanks for reviewing.

This might be a good situation to use the REST API directly to create diagrams and tiles in bulk. If you have some orchestration (or CloudFormation template) building the the channel resources, another program (Lambda) could use the ARNs from the channels resources to create a diagram and tile. The reverse could happen when the channel resources are deleted.
This document can help with that.
https://github.com/aws-solutions/aws-media-services-application-mapper/blob/main/docs/REST_API.md

Thank you for the document resource and guidance. Let me review this possibility.

Let us know if you need any recommendations on using the REST API for creating diagrams, tiles, or subscribing alarms to visual resources in diagrams.


charts: fix reference to container security context defaults

Fixed in https://github.com/ariga/atlas-operator/pull/173/files


visible environment variables

see column #3 "${env:THE_AUTHOR}"

this issue fixed in version 4.0
https://github.com/HandyOrg/HandyWinGet/releases/tag/v4.0.0-beta.1


Python 3.8 -- YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated

Is a warning showing on run when I run this via python 3.8.1

Yes. same problem with python 3.7.
The problem is located in main.py, l.23:
data = json.load(data_file)
One can solve it by replacing  l.23 by this line:
data = json.safe_load(data_file)
Details:
To understand this issue have a look at the import line 7 in main.py:
import yaml as json
and have a look at this documentation:
https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation


add macOS mount flags

#1192 add macOS mount flags according to
https://github.com/apple/darwin-xnu/blob/master/bsd/sys/mount.h#L288

You can remove bitrig.  It's dead, and even libc doesn't support it any more.  Also, you should add a note in CHANGELOG.


Many fibers with LOWEFFTIME in petal 9 for TILEID = 3297

I marked as unsure tile 3297 as it presents a lot of fibers with LOWEFFTIME accumulated in petal 9.

This is usually because the calibration stars preferentially sample areas where the fiber positioning is poor, leading to a low THRUFRAC and therefore low EFFTIME.  These are okay to mark as good.


What is the use case of xnd over apache arrow?

Hello, was wondering what you plan on doing with xnd that isn't well supported by the arrow format. A github issue is probably not the best place for this discussion, but I couldn't find a mailing list. Thanks.

Primarily any container that is described by datashape which is more
general than arrow --- as far as I am aware.
We will focus initially on tensor use-cases and look to be compatible in
areas of overlap.
Xnd is a general purpose meta-container more than a single container.
Travis
On Mar 7, 2018 5:19 PM, "Naveen Michaud-Agrawal"<EMAIL_ADDRESS>wrote:

Hello, was wondering what you plan on doing with xnd that isn't well
supported by the arrow format. A github issue is probably not the best
place for this discussion, but I couldn't find a mailing list. Thanks.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/plures/xnd/issues/6, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPjoDfrgN_8J5rJoRFYdTEkDIHbig91ks5tb_fdgaJpZM4SgnHR
.

Ah ok, it seems most of the underlying types and structured types (lists, structs, ragged hierarchies) are already well supported in arrow. Anyway looking at the docs it should be pretty cheap to convert from one format to another through memory copying.

Are they?  I though Arrow was limited to int32_t in sizes.  Xnd is for in-memory computations, so we definitely need int64_t.
Also, the types that xnd uses (ndt_t) are a standard algebraic datatype that is relatively easy to use for traversing memory, which is needed in gumath.  There is no dependency on an external C++ library like flatbuffers.

That said, we might translate Arrow to ndt_t in the future, but it is not a immediate priority (gumath is).

Are they? I thought Arrow was limited to int32_t in sizes. Xnd is for in-memory computations, so we definitely need int64_t.

This isn't quite true, see https://issues.apache.org/jira/browse/ARROW-750 -- support for very large variable-length collections is something we will eventually need to add to the format whenever there is demand for it.
In general, datasets will not be expected to be in a contiguous columnar memory block, but instead split across a collection of smaller chunks. We have discussed the 32- vs 64-byte issue for encoding collection lengths and the consensus has been that it is not worth the extra 4 bytes of overhead per value when the "large collection" case represents a very small percentage of use cases.

Additionally, we have changed 1-dimensional array sizes to use int64 almost a year ago https://github.com/apache/arrow/commit/ced9d766d70e84c4d0542c6f5d9bd57faf10781d#diff-520b20e87eb508faa3cc7aa9855030d7

On Wed, Mar 07, 2018 at 07:49:25PM +0000, Wes McKinney wrote:

Additionally, we have changed 1-dimensional array sizes to use int64 almost a year ago https://github.com/apache/arrow/commit/ced9d766d70e84c4d0542c6f5d9bd57faf10781d#diff-520b20e87eb508faa3cc7aa9855030d7

Thanks, I think I looked at the Arrow layout shortly before that date.  We still need n-dimensional large fixed arrays though.
BTW, our ragged array type also uses int32_t, and is Arrow compatible in the offset and bitmap layout. I agree that it makes sense to use int32_t for the offsets, since they can get quite large.

Makes sense. In our experience, Tensors are a different beast and use case from structured columnar data, so we are handling ndarrays / tensors with metadata separate from 1D record batches: https://github.com/apache/arrow/blob/master/format/Tensor.fbs#L35. These use 64-bit shape and strides. This is used actively by the Ray project

Ideally we can convert without memory copying.  I have not seen that arrow
is general enough.
Ndtypes should support an arrow memory container.  Arrow does not support
general ndtypes descriptors.
Arrow is a generalization of pandas.  Xnd is a generalization of pandas.
Arrow could use libxnd for functionality.
There is overlap but the tradeoffs are quite different.
Happy to be shown how the vision of xnd could be accomplished with just
arrow.  But I haven't seen that yet.
On Mar 7, 2018 8:08 PM, "Naveen Michaud-Agrawal"<EMAIL_ADDRESS>wrote:

Ah ok, it seems most of the underlying types and structured types (lists,
structs, ragged hierarchies) are already well supported in arrow. Anyway
looking at the docs it should be pretty cheap to convert from one format to
another through memory copying.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/plures/xnd/issues/6#issuecomment-371225023, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAPjoD9AiYjW59ku_oe0cFcp03xkuO8gks5tcB7xgaJpZM4SgnHR
.

There is overlap but the tradeoffs are quite different.

Agreed. We should look for opportunities to share code and infrastructure where possible. Note that the Arrow columnar format is but one type of data structure that we support -- it's a very important one by databases, Spark, pandas, etc. In order to implement zero-overhead memory sharing for structured datasets, many lower-levels of platform tooling must be created. I want to make sure we don't miss out on the collaboration opportunities for not having agreed on a "universal" data structure. The Arrow columnar format was never intended to be a universal data structure.

I agree it would be nice to have a standard low-level data structure.  For C, Ndtypes is pretty standard:  It describes all basic C types (including nested types, pointer types) using a regular algebraic data type.  One could use it e.g. for the type part in "Modern Compiler Implementation in C" (Appel et al.) without changes.
The tagged union convention is also the same as in the quoted book, and incidentally also the same as in Python's own compiler (whose author probably also read Appel, given that he used ASDL to describe the AST :).
I think columnar data can be modeled in ndtypes as a record of arrays. The example from the Arrow home page:
>>> data = {'session_id': [1331247700,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>],
...         'timestamp': [1515529735.4895875,<PHONE_NUMBER>.2128427,<PHONE_NUMBER>.4485607,<PHONE_NUMBER>.2181058],
...         'source_ip': ['<IP_ADDRESS>', '<IP_ADDRESS>', '<IP_ADDRESS>', '<IP_ADDRESS>']}
x = xnd(data)
>>> x.type
ndt("{session_id : 4 * int64, timestamp : 4 * float64, source_ip : 4 * string}")

There is categorical data, the representation of which is an array of indices into the categories:
>>> levels = ['January', 'August', 'December', None]
>>> x = xnd(['January', 'January', None, 'December', 'August', 'December', 'December'], levels=levels)
>>> x.value
['January', 'January', None, 'December', 'August', 'December', 'December']
>>> x.type
ndt("7 * categorical('January', 'August', 'December', NA)")

There are nested tuples, which are more general than ragged arrays:
>>> unbalanced_tree = (((1.0, 2.0), (3.0)), 4.0, ((5.0, 6.0, 7.0), ()))
>>> x = xnd(unbalanced_tree)
>>> x.value
(((1.0, 2.0), 3.0), 4.0, ((5.0, 6.0, 7.0), ()))
>>> x.type
ndt("(((float64, float64), float64), float64, ((float64, float64, float64), ()))")
>>>
>>> x[0]
xnd(((1.0, 2.0), 3.0), type="((float64, float64), float64)")
>>> x[0][0]
xnd((1.0, 2.0), type="(float64, float64)")

In general, xnd just takes any basic Python value -- nested or not -- and unpacks
it to typed memory.

[Repost because of broken markdown in email replies.]
I agree it would be nice to have a standard low-level data structure.  For C, Ndtypes is pretty standard:  It describes all basic C types (including nested types, pointer types) using a regular algebraic data type.  One could use it e.g. for the type part in "Modern Compiler Implementation in C" (Appel et al.) without changes.
The tagged union convention is also the same as in the quoted book, and incidentally also the same as in Python's own compiler (whose author probably also read Appel, given that he used ASDL to describe the AST :).
I think columnar data can be modeled in ndtypes as a record of arrays. The example from the Arrow home page:
>>> data = {'session_id': [1331247700,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>],
...         'timestamp': [1515529735.4895875,<PHONE_NUMBER>.2128427,<PHONE_NUMBER>.4485607,<PHONE_NUMBER>.2181058],
...         'source_ip': ['<IP_ADDRESS>', '<IP_ADDRESS>', '<IP_ADDRESS>', '<IP_ADDRESS>']}
x = xnd(data)
>>> x.type
ndt("{session_id : 4 * int64, timestamp : 4 * float64, source_ip : 4 * string}")

There is categorical data, the representation of which is an array of indices into the categories:
>>> levels = ['January', 'August', 'December', None]
>>> x = xnd(['January', 'January', None, 'December', 'August', 'December', 'December'], levels=levels)
>>> x.value
['January', 'January', None, 'December', 'August', 'December', 'December']
>>> x.type
ndt("7 * categorical('January', 'August', 'December', NA)")

There are nested tuples, which are more general than ragged arrays:
>>> unbalanced_tree = (((1.0, 2.0), (3.0)), 4.0, ((5.0, 6.0, 7.0), ()))
>>> x = xnd(unbalanced_tree)
>>> x.value
(((1.0, 2.0), 3.0), 4.0, ((5.0, 6.0, 7.0), ()))
>>> x.type
ndt("(((float64, float64), float64), float64, ((float64, float64, float64), ()))")
>>>
>>> x[0]
xnd(((1.0, 2.0), 3.0), type="((float64, float64), float64)")
>>> x[0][0]
xnd((1.0, 2.0), type="(float64, float64)")

In general, xnd just takes any basic Python value -- nested or not -- and unpacks
it to typed memory.

I am skeptical about the idea of an all-powerful / can-describe-anything data structure. With generalization comes added complexity for computational frameworks and producers/consumers.
@teoliphant stated "I have not seen that arrow is general enough." What does this mean? At this point, the Arrow columnar format is only one part of a much larger project. I think this means "the Arrow columnar format is not a universal data structure", which I agree with, but that was never the goal. I see the work here in libndtypes / xnd as complementary and not in conflict -- there are problems being solved (extending the notion of NumPy's structured dtypes to support things like variable-length cells and pointers) that were never in scope for Arrow's columnar format.
The columnar format was the focus of the project at the outset because that was the most immediate and high value problem to solve around data interoperability and in-memory analytics. The rapid uptake of the project and developer community growth suggests we made a good bet on this.
At this point Arrow a multi-layered project of memory management, shared memory (Plasma), metadata serialization, IO, streaming messaging, memory formats (including the columnar format), file format interop, computation kernels, etc. The work that is being done here could even become an additional component of Apache Arrow if you wanted to work with a larger developer community. At minimum it would be helpful to have a broader design/architecture discourse about problems and use cases in a public venue.

Ideally we can convert without memory copying.

As demonstrated in ArrayViews, one can wrap Arrow arrays with xnd, and vice versa, without memory copying. However, currently the wrapping does not support null buffers (Arrow) or bitmaps (xnd) because xnd does not expose bitmaps.

@pearu the cases where the memory is compatible IMHO reflect a minority (and a small minority at that) of real world use of Arrow. To suggest "compatible, with some exceptions" will mislead people

@wesm, I am not sure that I follow your comments meaning. If you refer to the fact xnd does not expose bitmaps, then this issue can be easily fixed as xnd bitmap is compatible with Arrow null buffer. I guess the reason of not exposing xnd bitmaps is that it is consider as internal structure while in Arrow null buffer is not that.

You stated

As demonstrated in ArrayViews, one can wrap Arrow arrays with xnd, and vice versa, without memory copying

I think it's worth making a list of different Arrow use cases:

Primitive (C-like) arrays with no nulls
Primitive arrays with nulls
Varbinary / utf8 (variable-length) arrays with nulls
Nested type arrays (e.g. list<binary> or list<int32>)
Unions

Do you support them all and export all of their semantics in xnd? If the answer is "no", then I think you need to qualify the statement to say that "In certain limited cases, one can wrap Arrow arrays [and expose their semantics], without memory copying"


PURE训练后进行预测提示错误

训练后进行预测出现问题

找不到这个包，提示报错。

Traceback (most recent call last):
  File "predict.py", line 11, in <module>
    from deepke.triple_extraction.PURE.module.data_structures import Dataset, evaluate_predictions
ModuleNotFoundError: No module named 'deepke.triple_extraction.PURE.module'

Environment:

OS: [Linux]
Python Version [e.g. 3.7]
deepke                   2.2.7        /root/miniconda3/lib/python3.7/site-packages/deepke-2.2.7-py3.7.egg

我看了一下好像关于PURE的包都没了，看GITHUB的代码里面也没有，这个库被遗弃了吗？

I'm sorry for the inconvenience. It seems that there was a mistake in our code. To fix this issue, you can try adding an 's' after the word 'module', making it 'modules.data_structures'. Please try this solution and let us know if it works. If you encounter any further difficulties, don't hesitate to ask for help. Thank you for your understanding and patience!

I'm sorry for the inconvenience. It seems that there was a mistake in our code. To fix this issue, you can try adding an 's' after the word 'module', making it 'modules.data_structures'. Please try this solution and let us know if it works. If you encounter any further difficulties, don't hesitate to ask for help. Thank you for your understanding and patience!

I tried your method, but it may not be the cause of the bug. Upon inspecting the source code, I found that there were corresponding methods for ASP and PRGC in the triple extraction, but the PURE file was empty.

Sorry, all the code for PURE has been moved to the Example file before, and that part of the code may need to be modified. The location of the package is in the DeepKE/example/triple/PURE/models file.

You can try changing deepke.triple_extraction.PURE.module.data_structures to models.data_structures. If you have any further questions or need further assistance, please feel free to let me know. Thank you for your patience!

Thanks,I will try it.


Possible issue with RADIO_STICK_MIN RADIO_STICK_MAX

I think we need to check RADIO_STICK_MIN RADIO_STICK_MAX for use with motor trimmers.
@gunturiCM  had mentioned that previously (before #52) the throttleReferencePercentage would sometimes go negative, I had checked the RADIO_STICK_MIN RADIO_STICK_MAX when the motor trimmers where set to zero and they where correct so couldn't figure out why it would go negative.
Just realised, when we use the trimmers to balance the quadrotor to hover properly I think it shifts the range of values from the RC therefore getting negative values or not getting top speed from the motors (depending how it shifts)
Think we should run hw_test_receiver.py on the Pi to see what the RADIO_STICK_MIN RADIO_STICK_MAX are when the trimmers are at there most extreme. Obviously by increasing RADIO_STICK_MIN and decreasing RADIO_STICK_MAX it will be slightly hard to control but we can see how far we usually use the trimmers and stay close to them values.
I realise @gunturiCM has sorted the issue of it going negative but we should still fix this to be able to use the full range of the motor at all times.

I think we can have a calibration step in which we prompt the user to shift the throttle to min and max positions (including trimmers) and record them in a file on R-Pi. We can use these stored values for every (say) 100 runs and then we ask the user to calibrate again. The disadvantage is that if the user changes the trim in the midst of a flight, these stored values become irrelevant. Some flight controllers (like the Pixhawk if I remember correctly) have this step as mandatory on the first run.

I think it will just be the max and min for the pitch and roll that will change, so we could keep RADIO_STICK_MAX RADIO_STICK_MIN as they are for all others (throttle and trimmers A-D) and have new variables something like PITCH_RADIO_STICK_MAX PITCH_RADIO_STICK_MIN , ROLL_RADIO_STICK_MAX ROLL_RADIO_STICK_MIN.
@alphaville and I were talking about this the other day before the meeting and we thought rather than the user having to calibrate ever so often, if we just take the values first sent form the RC's pitch and roll channel as the centre point and add 700 for the PITCH_RADIO_STICK_MAX, ROLL_RADIO_STICK_MAX and subtract 700 for the PITCH_RADIO_STICK_MIN, ROLL_RADIO_STICK_MIN. This means the first time the user fly's the quadrotor the can use the trimmers to balance the quadrotor to hover properly and then after these trimmers are set they can reset the drone.

Yes! I agree, we can do this.

Folks, I'm leaving this to you to sort out
@jamie-54 @gunturiCM

Sure sir.


"Cannot read properties of null" when using trim with nullish

This code throws TypeError: Cannot read properties of null (reading 'trim')
const Schema = v.object({
  foo: v.pipe(v.nullish(v.string()), v.trim()),
});

const result = v.safeParse(Schema, {
  foo: null,
});

Is there any reason we're not using safe navigation operator when applying transforms?
e.g.
dataset.value = dataset.value?.trim();

This is related to #669. There should be a TypeScript warning to prevent this from happening when using TypeScript.
The fix is to wrap nullish around pipe. This way only strings will pass through the pipeline. See this playground.
import * as v from 'valibot';

const Schema = v.object({
  foo: v.nullish(v.pipe(v.string(), v.trim())),
});
``

I see, thank you. Definitely worth to add this to the documentation.

I think the current approach is simple and clear. You can't trim null, but you can trim string. So it makes sense to only pipe the string schema. Changing this behavior would, in my opinion, make it less clear how to use our API. However, I appreciate your feedback and will look into it again when I take a closer look at #669.

There will now be a TS warning in the latest version if our API is used incorrectly.


"Intra subnet traffic"

Please can you clarify the guidance for Intra-subnet traffic?  The statement below seems to suggest that traffic between hosts within a subnet actually traverses the subnet boundary for it to be affected by the NSG?  It makes perfect sense for security rules in an NSG applied to a NIC, not so much sense (to me) that security rules in an NSG at the subnet boundary should do this.
"It's important to note that security rules in an NSG associated to a subnet can affect connectivity between VM's within it. For example, if a rule is added to NSG1 which denies all inbound and outbound traffic, VM1 and VM2 will no longer be able to communicate with each other. Another rule would have to be added specifically to allow this."

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: cd190c21-5b50-87ba-37dd-0b0c017ec6fb
Version Independent ID: d9720147-207c-4c98-ba8d-016ff4b6ad60
Content: Network security group - how it works
Content Source: articles/virtual-network/network-security-group-how-it-works.md
Service: virtual-network
GitHub Login: @KumudD
Microsoft Alias: kumud

@deandownunder , thank you for your query. We'll review this and get back to you shortly!

@deandownunder , I'm following up on my previous comment. Kindly let us know if the above helps or you need further assistance on this issue.

@deandownunder , We are closing this issue for now. If there are further questions regarding this matter, please reply and we will gladly continue the discussion.


Fix TypeScript def for ComponentProps generic. Fixes #41

In case this is desired. If it should be something other than <any> happy to change.

It can be more specific and use this instead of any

Done.

Is there a workaround for issue https://github.com/developit/preact-mdl/issues/41 when using preact-mdl with TypeScript? If not, then just wondering if this PR could be accepted to be able to work with TS and mdl in the same project, thanks. cc @developit


Rename and authorization for namespaced expand-spec endpoint

What this PR does / why we need it:
The expand-spec endpoint is renamed to expand-vm-spec and made
namespaced to allow proper authorization.
This PR adds code to the virt-api authorizer to check namespaced
base resource endpoints and unittests/functests to verify the authorization.
Based on https://github.com/kubevirt/kubevirt/pull/8399 and https://github.com/kubevirt/kubevirt/pull/8570
/area instancetype
Release note:

The expand-spec subresource endpoint was renamed to expand-vm-spec and made namespaced

Looks good.
Have you considered renaming the endpoint to something that would hint that it takes a VM definition, like expand-vm-spec?
It would be harder to change name once the API is merged.
/lgtm

/retest

@akrejcir Good idea, I renamed it to expand-vm-spec

/lgtm

@akrejcir I had to rebase, can you lgtm again?

/retest

/lgtm

/retest

/retest

@davidvossel I updated the PR and tried to keep the changes to the authorizer minimal. Can you PTAL again?

@lyarwood I addressed your comments, can you PTAL?

@lyarwood I addressed your comments, can you PTAL?

/lgtm
Thanks changes look good!


Add wait_response argument

Resolves #12

Codecov Report

Merging #13 (2548097) into master (6edb8c4) will not change coverage.
The diff coverage is 100.00%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff            @@
##            master       #13   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           20        20
  Lines          408       413    +5
=========================================
+ Hits           408       413    +5

Impacted Files
Coverage Δ

aioarp/_client.py
100.00% <ø> (ø)

aioarp/_async.py
100.00% <100.00%> (ø)

aioarp/_sync.py
100.00% <100.00%> (ø)

tests/test_async.py
100.00% <100.00%> (ø)

tests/test_sync.py
100.00% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Multidimensional input support? (e.g. CNN channels)

Not sure if this is supported?  Mind clarifying, and a quick example of how to structure an example input?
Thanks!!

At the moment, the input layer is just an FC layer which does not carry out any summing/activations. Input therefore needs to be flattened, into a one dimensional array.
For example (keeping MNIST in mind, still), this is an example of a (very unoptimized) conv net, where the 28x28 images are passed in as 784 long arrays:
net = new Network({
    Module: Module,
    layers: [
        new FCLayer(784),
        new ConvLayer(1, {filterSize: 5, zeroPadding: 2, stride: 1, activation: "lrelu"}),
        new FCLayer(784, {activation: "sigmoid"}),
        new FCLayer(10)
    ],
    learningRate: 0.01,
    lreluSlope: -0.0005,
    l2: true
})

This is due to jsNet originally only implementing MLPs. Would you prefer a way to give input as a 2d or 3d array?

I mean, sure, if it could do matrix operations on multidimensional, that would be great.  What about the channels option you have for Network initialization?  I assumed that maybe allowed for at least a 784x3 input shape (if flattening the 28x28 as you suggest).

In the MNIST example, the images happen to be grayscale, so it's only 1 channel. You'd be right in thinking that channels would be set to 3 for RGB images, where there's 3 channels.
And ok, I'll add volume as input to my TODO list. Working on v4.0, but with this, there may be enough for a v3.3.

I've committed an initial implementation for this. You should now be able to pass in input data as a volume. You can try this out on the dev branch, for now, if you'd like. There's also an InputLayer now, which might optionally make the input layer clearer to set up (docs updated).
If you were going to use this for a particular public data set, let me know, and I could whip up some examples/demo files.

Sounds great - I'll download and check out.  Regarding the ConvLayer... convolutional layers are 3D by nature, no?  HxW with depth being the channels.  Likewise the 3D filter (filter size in HxW with matching depth of the # channels) shifts over the 2D grid, to convolv the matrix.  So how does your convolution work if in 1D form?
I'm actually interested in using this for multivariate time series analysis (diff variables being in diff channels), so ideally inputting for me would be something like:
// number rows = number "channels"
[
  [a1, a2, a3... aN],
  [b1, b2, b3... bN],
  [c1, c2, c3... cN],
  [d1, d2, d3... dN],
  ...
]

Also, and slightly unrelated, I'm under the impression that a single value output defaults to logistic or sigmoid or something, and any array defaults to softmax.  That would work for most cases, but do you have the ability to output a vector or matrix of say, logistic activation?  This would certainly be handy in multivariate regression, for instance.

Thanks again!

K, so I don't quite follow how your multi-dimension syntax works for defining the InputLayer
you have:
new InputLayer(3, { span: 5 }) // 75 inputs

How does that work exactly?
Other libraries usually define an input "shape" vector or something... like [3,2,10] with each scalar pertaining to a dimension.
Could we parse an array as the multidimensional shape if the first param? (rather than the implied 1D input shape if an integer)

Ah, I fixed the readme for the InputLayer. I think I mis-copy/pasted from dillinger. Let me know if changing it to a shape vector makes more sense. It would likely have just 2 elements, as filters are square, so the height/width is the same (so depth, and span). I do hope one day to get around to re-implementing the filters to work with different X and Y sizes, but right now it's just squares.
I removed the implicit softmax on the last layer as well. I've been meaning to change this for a while. It's now set via a {softmax: true} config in OutputLayer, returning the activations, otherwise. It also means that an activation function can be enabled/disabled for the last layer.
Regarding the 1D conv question, just to clarify, do you mean when the input map (across however many channels) is 1 x 1 'pixels'? If so, I haven't tested any such data sets, but keeping zero padding at 0 and filterSize at 1, this in theory should just do a dot product of the input values across channels, against the filter weights (one per channel). Is this your question, and if so, the expected behaviour?
Thanks for the feedback. Do let me know of things that need changing.

Sorry it's taken so long to get to this, but about to dive in on the CNN testing now.  With my data, I'm thinking multichannel 1D convolution would be best.  Basically feed the time series window in as 1D arrays, but with each variable as separate channels.  Would this be the best way to handle this?
Based on your docs, it appears that if I were to define multiple channels, the second "span" param implies the square dimension of a 2D grid.  Any way to have multichannel 1D?
Secondly, assuming 1D multichannel support, how would you form the input data?  Would it be like this (for a 2 channel, 128 width 1D convolution):
let example = {
  input: [
    [ 0, 1, 0, .. 1 ], // channel 1 = 128 width array
    [ 1, 1, 0, .. 0 ], // channel 2 = 128 width array
  ],
  expected: [ 0, 1 ] // or whatever output
}

Thanks!!

Having read more about 1D conv (I haven't personally used it before), I don't think that it would work with jsNet. At the moment, all filters and conv input are squares. If my understanding is correct, a 1D conv would need the filters and inputs to be a 1D array.
For this to work, there'd probably need to be a Conv1DLayer (and maybe Filter1D), or something similar, like in some other frameworks I've seen do this.


Selecting Pinned Columns

Adding the ability to select pinned columns like any regular column with Ctrl mouse click.

One more use case to cover -- users should be able to copy the selected data to the clipboard (select some pinned columns & some rows) . This doesn't seem to work with pinned columns (tried the same selection on pinned/regular columns, copy-paste works only on regular columns)

The functionality to copy values from pinned columns was added in the version 1.1.17 of PowerGrid package

@dgosi please take a look! Can't properly select and copy data on pinned column. I'm testing on PowerGrid package 1.1.19


_MklSoftmax 2-2.5x Slower in 1.15 Compared to 1.14 and 1.13

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): intelaipg/intel-optimized-tensorflow:1.14.0-mkl-py3 and intelaipg/intel-optimized-tensorflow:1.15.2-mkl-py3
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: n/a
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): see OS
Python version: 3.6
Bazel version (if compiling from source): n/a
GCC/Compiler version (if compiling from source): n/a
CUDA/cuDNN version: n/a
GPU model and memory: n/a

Describe the current behavior
We found that the _MklSoftmax operation is quite a bit slower in 1.15 than it was in Tf 1.13 and 1.14, about 2-2.5x worse.
Describe the expected behavior
Comparable speed to previous versions.
Standalone code to reproduce the issue
n/a but confirmed by @NeoZhangJianyu (see https://github.com/tensorflow/tensorflow/issues/39851#issuecomment-652150250)
Other info / logs
n/a

@pks
Will it be possible to share colab link or simple standalone code to reproduce the issue in our environment.It helps us in localizing the issue faster.Thanks!

We found that in our implementation of a Transformer model, which I unfortunately cannot share. @NeoZhangJianyu said that he could reproduce it. He could possibly add a minimal example.

@ravikyram @pks
Yes, I have reproduced the issue.
I have reported this issue to dev team. The checking is ongoing.
Thank you!

@NeoZhangJianyu Is there any progress on this?

@pks It's still on going. I will push to confirm and fix.

@pks
We got the answer from dev team:
The TF 1.15 uses the mkldnn 0.20 version, which will run new path (avx512 impl on CLX or SKX), while TF 1.14 with 0.18 mkldnn will run a reference implementation.
The new path will take much more time for primitive creating than the latter path.
It's the root cause why we have poor performance for SoftMax on TF 1.15, especially for small problem size.
And TF master branch has enabled the primitive cache for SoftMax, so there's no such performance issue.
This issue has been fixed in TF 2.2.0 and later. Please use TF 2.2.0 and later.
We can't port this new feature back to TF 1.15 because Google has cut off the updates for that branch.

Thanks a lot for the investigation! Closing this issue for now.

@NeoZhangJianyu That's unfortunate, since the softmax is a pretty central operation in neural networks (especially Transformers).
What about machines that do not have AVX-512 extensions (Haswell?), will those run the reference implementation? And is there a way to enable the primitive cache when building Tensorflow from source? Switching to Tensorflow 2 is a longer endeavor if one wants to directly run graphs.

@pks
For the machine without AVX-512, the issue is present too: it uses AVX2.
I will check for any work around and feedback later.
Additional, it's recommended to upgrade to Tf 2.x.
For TF 1.15: This is the last 1.x release for TensorFlow. We do not expect to update the 1.x branch with features, although we will issue patch releases to fix vulnerabilities for at least one year. -- refer to https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md

@pks
I checked with the designer. There is no simple way to merge back the solution from TF 2.x to TF 1.15.

@NeoZhangJianyu Alright, thanks for checking!

@pks There is solution for this issue now.
Intel release a special version: Intel TF 1.15.0up1 today. It includes many bugs fix of TF2.x.
I have tested the MKLSoftmax issue. The performance is increased than 1.15 in small size (like 1000).  Note, all of releases have same performance in big size (like 100,000).
But it is still not same as TF 1.14.
TF 2.2.0-MKL is with same performance of TF 1.14.
Please build it from source code by following guide:
$ git clone https://github.com/Intel-tensorflow/tensorflow.git
$ git checkout v1.15.0up1

bazel build --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0 --copt=-O3 --copt=-Wformat --copt=-Wformat-security --copt=-fstack-protector --copt=-fPIC --copt=-fpic --linkopt=-znoexecstack --linkopt=-zrelro --linkopt=-znow --linkopt=-fstack-protector --config=mkl --define build_with_mkl_dnn_v1_only=true --copt=-DENABLE_INTEL_MKL_BFLOAT16 --copt=-march=native //tensorflow/tools/pip_package:build_pip_package

./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

pip install /tmp/tensorflow_pkg/tensorflow-xxx-xxx.whl

Thanks @NeoZhangJianyu! Unfortunately I don't seem to be able to build the wheel from this branch:
> bazel build --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0 --copt=-O3 --copt=-Wformat --copt=-Wformat-security --copt=-fstack-protector --copt=-fPIC --copt=-fpic --linkopt=-znoexecstack --linkopt=-zrelro --linkopt=-znow --linkopt=-fstack-protector --config=mkl --define build_with_mkl_dnn_v1_only=true --copt=-DENABLE_INTEL_MKL_BFLOAT16 --copt=-march=native //tensorflow/tools/pip_package:build_pip_package

Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=127
INFO: Reading rc options for 'build' from ~/intel_tensorflow/.bazelrc:
  'build' options: --apple_platform_type=macos --define framework_shared_object=true --define open_source_build=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone --strategy=Genrule=standalone -c opt --announce_rc --define=grpc_no_ares=true --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include
INFO: Found applicable config definition build:mkl in file ~/intel_tensorflow/.bazelrc: --define=build_with_mkl=true --define=enable_mkl=true --define=tensorflow_mkldnn_contraction_kernel=0 --define=build_with_mkl_dnn_v1_only=true --copt=-DENABLE_INTEL_MKL_BFLOAT16 -c opt
WARNING: ~/intel_tensorflow/tensorflow/core/BUILD:1091:11: in linkstatic attribute of cc_library rule //tensorflow/core:framework_lite: setting 'linkstatic=1' is recommended if there are no object files
WARNING: ~/intel_tensorflow/tensorflow/core/BUILD:2428:11: in linkstatic attribute of cc_library rule //tensorflow/core:lib_internal: setting 'linkstatic=1' is recommended if there are no object files
ERROR: ~/intel_tensorflow/tensorflow/core/common_runtime/eager/BUILD:277:11: in nocopts attribute of cc_library rule //tensorflow/core/common_runtime/eager:mkl_eager_op_rewrite: This attribute was removed. See https://github.com/bazelbuild/bazel/issues/8706 for details.
ERROR: ~/tf115/intel_tensorflow/tensorflow/core/common_runtime/eager/BUILD:277:11: in nocopts attribute of cc_library rule //tensorflow/core/common_runtime/eager:mkl_eager_op_rewrite: This attribute was removed. See https://github.com/bazelbuild/bazel/issues/8706 for details.
ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted: Analysis failed
INFO: Elapsed time: 13.751s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (402 packages loaded, 17775 targets configured)
    Fetching @llvm; Restarting.

This seems to be a problem with bazel. Which version should I be using?
> bazel --version
bazel 3.4.1

@pks My bazel is 3.1.0.
There is binary release now, please install the TF 1.15.0up1 by PIP:
pip install https://storage.googleapis.com/intel-optimized-tensorflow/intel_tensorflow-1.15.0up1-cp36-cp36m-manylinux2010_x86_64.whl

@pks It's great news! 👍
Thank you support!

@NeoZhangJianyu Which is the earliest CPU generation the wheel will work with?

@pks
It will work well with the CPU supports AVX2 & AVX512. AVX512 could speed up more.
The info could be searched in https://ark.intel.com/content/www/us/en/ark.html#@Processors

@pks
Hi,
It's great to know TF 1.15.0up1 to fix your issue.
I want to collect the customer to expect/use TF 1.15.0up1.
Could you share your company name to me?
You could send to my email<EMAIL_ADDRESS>Thank you!


Hot-reloading for Vite dev server

Issue

Currently we use npm run start, which uses vite to run the dev server.
A prestart script is used to run tsc so that TS modules are compiled to JS before the server starts.
Ideally we would have hot-reloading, so that changes to the components would reflect on the running demo page.

Possible Solutions

Ideally the Vite configuration can handle this somehow. I assume it's not picking up the .ts files as the .js files are referenced everywhere.
A package like npm-run-all or concurrently (or pnpm) to run multiple scripts at once.
a. We could have a script to run tsc --watch and another for vite to run together.
Something else!

Vite itself does not require compilation. The problem is Polymer analyser which only understands JS.

Something else!

I'm 90% complete replacing that whole setup with Storybook.

Nice, sounds ideal!


Allow uploading empty files

I'm now aware of a reason why we shouldn't be allowing 0-sized uploads.

Hey, thanks!
Empty files are indeed something one might want to upload. For example, to retain empty directories. Gentoo's Portage uses this, and git would ignore an empty directory (therefore files like .keep are kept in places).
I would merge this, but it needs a test. A suggestion: Copy an existing one, supply an empty string to strings.NewReader (or something else), set Content-Length accordingly. If you know a better way to write a minimal test, all the better.

Sure, I just pushed a test. The compareContents utility function was acting up on the empty file, so I used os.Stat instead.
Also, please note that I see two upload_test.go failures in master (OSX High Sierra). I haven't fixed them to keep the diff small.

Done.

Awesome. Thanks!

If nobody vetoes it, I will push a new revision of this plugin, with your works included, by the end of the week.


Refactor ServerServiceDefinition and move it to Grpc.Core.Api nuget.

Followup for #17778

change ServerServiceDefinition so that it no longer references internal Grpc.Core types.  Instead, it stores lambdas that are later used to invoke AddMethod<,>() calls on ServiceBinderBase (the advantage is that it preserves generic context)
move ServerServiceDefinition and ServiceBinderBase
refactor Interceptor code as the new ServerServiceDefinition makes intercepting the handlers easier.

Known failures: https://github.com/grpc/grpc/issues/17561

@JunTaoLuo @JamesNK I believe the review comments have been addressed sufficiently, so I'm going to merge now. If there are still any concerns, we can address in a followup PR.

One suggestion if you could swap out the dictionary for a hashset of string. It’s not important though.

@JamesNK   I considered that, but HashSet.Add doesn't throw if the key already exists, so using dictionary.Add seemed safer in terms of preserving the old behavior.

Not sure if we can access your link without an @google.com login. We'll just wait for the packages to be available on packages.grpc.io.

Not sure if we can access your link without an @google.com login. We'll just wait for the packages to be available on packages.grpc.io.

The link is public, so you should be able to just download.
But packages.grpc.io  already has the nugets: https://packages.grpc.io/archive/2019/02/977df7208a6e3f9a62a6369af5cd6e4b69b4fdec-a9f2947c-fd01-4598-9af0-93e276d53335/index.xml


Hex encoded literal has bad AST

If I have the following test:
cast2.c
#include "tests.h"

typedef unsigned int pcre_uint32;
#define CHAR_NBSP                   ((unsigned char)'\xa0')

void test_cast()
{
    int tmp = 160;
    pcre_uint32 chr = tmp;

    is_eq(chr, CHAR_NBSP);
}

int main()
{
    plan(1);

    diag("Compare preprocessor with type")

    test_cast();

    done_testing();
}

I can execute it with clang -lm -o main cast2.c && ./main and obtain:
1..1
# Compare preprocessor with type
1 ok - chr == CHAR_NBSP

However with c2go (c2go transpile -o main.go -p main cast2.c && go run main.go) I obtain:
# command-line-arguments
./main.go:335:67: syntax error: unexpected %, expecting expression
./main.go:341:5: syntax error: unexpected ), expecting := or = or comma
./main.go:344:55: syntax error: unexpected newline, expecting { after if clause
./main.go:349:4: syntax error: unexpected else after top level declaration):

The problem is that CHAR_NBSP has been transpiled as (uint8(%!q(int=4294967200))).
While debugging I found that the AST node for it is:
|   | | |-ImplicitCastExpr 0x5583de4b9090 <col:24, col:48> 'double' <IntegralToFloating>
|   | | | `-ParenExpr 0x5583de508550 <col:24, col:48> 'unsigned char'
|   | | |   `-ParenExpr 0x5583de508530 <col:25, col:47> 'unsigned char'
|   | | |     `-CStyleCastExpr 0x5583de508508 <col:26, col:41> 'unsigned char' <IntegralCast>
|   | | |       `-CharacterLiteral 0x5583de5084e0 <col:41> 'int'<PHONE_NUMBER>

Is this a bug in clang?

@kamphaus yes, I see it would be more difficult - but nothing is impossible ;)
Unfortunately there are not real options for the clang AST, and there are several little bugs throughout (and across versions of clang). I think it's mainly used as a debugging tool for the parser which we are repurposing way beyond what it was originally intended for.


Feature/metube

Description of the change
Adds meetube to home ops, currently running via docker but exposed via cluster ingress.
Metube is gui application for yt-dls modern fork of youtbe-dl.

MegaLinter status: ✅ SUCCESS
See errors details in artifact MegaLinter reports on CI Job page
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff


Add files to set up basic firewall and routing along with DNS and DHCP

Some basic setup. There are some challenges around dealing with the network interface names and subnet addresses, like we discussed.

I see that the default OpenBSD vi and VSCode don't agree on tabs and spaces. :) :|


error after install

root@ubuntu:~/PlaystoreDownloader# python3 download.py "com.whatsapp"
Traceback (most recent call last):
  File "download.py", line 9, in <module>
    from playstore.playstore import Playstore
  File "/root/PlaystoreDownloader/playstore/playstore.py", line 9, in <module>
    import requests
ImportError: No module named 'requests'
root@ubuntu:~/PlaystoreDownloader# python3 download.py "com.facebook"
Traceback (most recent call last):
  File "download.py", line 9, in <module>
    from playstore.playstore import Playstore
  File "/root/PlaystoreDownloader/playstore/playstore.py", line 9, in <module>
    import requests
ImportError: No module named 'requests'
root@ubuntu:~/PlaystoreDownloader#

Hi, did you install the requirements pip3 install -r requirements.txt? It looks like a package wasn't installed correctly, did you follow all the instructions contained in the readme? Thanks.

it work now , thanks ,
but i have problem with login
root@ubuntu:~/PlaystoreDownloader# python3 download.py "com.facebook.lite" 29/10/2017 14:08:59> [CRITICAL][_login()] Login failed. Please check your credentials.
login details is correct and Allow less secure apps: ON :(

In order to avoid the download of the additional files you have to remove/comment this section from playstore/playstore.py:
https://github.com/ClaudiuGeorgiu/PlaystoreDownloader/blob/b0cea41f9d676279685bb66e6dca1401a830ed44/playstore/playstore.py#L404-L435
I will add a command line argument to make the additional files download optional, thanks for the suggestion.

thanks allot , i am waiting new update :+1:


Added VM test script

Added the script to be put into each VM to test Bareflank. Script located in hypervisor/tools/tests/BFTest.sh

Looks great. I will merge once the tests pass


Problem with landing page on iOS

Main page should read, "I am AN" in the case that client chooses "influencer"

I believe a(n) could work too!

I'll ping my in-house UX researcher and get this sorted ASAP 👍 Good catch

Thanks. I appreciate the quick response. If you're open to pull requests, I could take a crack at it 😄

Yes please do @srowhani! I'm swamped tackling a vital bug in the README

I believe the problem is the fact that 2+2 is 4, minus 1 that's 3 ... QUICK MATHS


Pace theme CSS file was not found: blue/pace-theme-minimal.css

Hello,
I have got the following error on executing the command sudo ember serve please help
error
`Pace theme CSS file was not found: blue/pace-theme-minimal.css
Stack Trace and Error Report: /tmp/error.dump.34ad42212756dbb722c91fdc465c7130.log
Pace theme CSS file was not found: blue/pace-theme-minimal.css
Stack Trace and Error Report: /tmp/error.dump.7c9947adb610b30193e9ee0070268dc9.log`

I got the solution, the problem was coming because I shifted the project folder to different location after installation and was using the command 'ember serve' but problem got solved when I shifted back and the project to initial location. One more thing I was not running my server which also lead to this problem.
Hope this helps!!


Unbound variable "PWD"

I followed the starting guide after cloning the repo.
yarn to install but got Unbound variable "PWD" when try yarn dev.
deleted the repo and reclone but still getting the same thing.  Any else got this ?

I am not able to replicate this error on a fresh clone

you can set it manually by typing
export PWD="$(pwd)"

hello, sorry I want to ask about the error "Unbound Variable 'PWD', where should I solve the problem. Thank you

windows,typing
$Env:PWD = Get-Location


[c++] Fix reindexer for when workload is small and threadpool is large

Issue and/or context: As discussed in Slack.
@bkmartinjr repro:
import numpy as np
import tiledbsoma as soma
from tiledbsoma import pytiledbsoma as clib

reindexer = clib.IntIndexer()
keys = np.arange(0, 10000, dtype=np.int64)
reindexer.map_locations(keys, 64)
ids = np.array([525, 1293, 1805, 5802, 7636, 7754, 7791, 7957, 7959, 8067, 8340, 8736, 8806, 9329,9377, 9653], dtype=np.int64)

# this hangs
reindexer.get_indexer(ids)

Changes:
Notes for Reviewer:

Duplicate of #2061 by @beroy


Add Kaggle 3 to Training & Outreach Overview page + nav bar menus

Add Kaggle 3 to Training & Outreach Overview page + nav bar menus
Please add links to Kaggle 3 to the HRA Portal.

https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature

This is an external link so it will open in a new tab. We will have a content page for K3 eventually when the competition is over.
To Do

[ ] Add K3 button card to Training & Outreach Overview
[ ] Add K3 menu item to Training & Outreach navigation menu on mobile/desktop.

Resources

Figma Design File: Training & Outreach Overview
Figma Design File: Desktop Navbar
Figma Design File: Mobile Navbar

Completed this in PR #233


Renew with bundled san certificates drops subject alternate names

hi there, just tried to renew my bundled certificate, but it always drops my SAN.
Tested with one --domains parameter that contained only the main certificate domain and also one test with all --domains parameter.
Tested versions: master and 0.2
Command to check:
openssl x509 -noout -text -in /etc/ssl/letsencrypt/certificate

Lego command:
./lego --exclude "tls-sni-01" --domains=blub.example.com -path=/etc/ssl/letsencrypt/<EMAIL_ADDRESS>--http <IP_ADDRESS>:9999 renew
Output:
2016/01/29 05:28:52 [INFO][blub.example.com] acme: Trying renewal with 1806 hours remaining
2016/01/29 05:28:53 [INFO][blub.example.com] acme: Obtaining bundled SAN certificate
2016/01/29 05:28:53 [INFO][blub.example.com] acme: Could not find solver for: dns-01
2016/01/29 05:28:53 [INFO][blub.example.com] acme: Trying to solve HTTP-01
2016/01/29 05:28:54 [INFO][blub.example.com] Served key authentication
2016/01/29 05:28:56 [INFO][blub.example.com] The server validated our request
2016/01/29 05:28:56 [INFO][blub.example.com] acme: Validations succeeded; requesting certificates
2016/01/29 05:29:05 [INFO] acme: Requesting issuer cert from https://acme-v01.api.letsencrypt.org/acme/issuer-cert
2016/01/29 05:29:05 [INFO][blub.example.com] Server responded with a certificate.

Please let me know if this commit does not solve your problem :)

will do. thx


plantuml.jar path can be set during package installation

Thanks, @Nikolai-Hlubek. Pulled and integrated into 0.0.3 (on pypi).


select2 treeview

This pull request includes a

[ ] Bug fix
[x] New feature
[ ] Translation

The following changes were made
select2 treeview #3269
example: select2-treeview
$("#test").select2({
treeview:true
});

commenting just to make sure I get notified when this is merged

This looks awesome!

I think this can be merged after it has tests and documentation.

@jpic
In fact, I'm not good at front-end development. If someone can help that is the best.

Sure, what's your question ?

What I mean is that everyone can modify my code, add related tests and documentation, then submit the PR. 😊

Can we leave it open meanwhile ? Perhaps someone will like to help.

Of course, select2 give US a lot of help, we want to make it better!


Shape based feature extraction

There are three files in the folder (polus-feature-extraction).

Feret Diamter.ipynb - This program is to calculate the feret diameter of an object. To calculate memory efficiently, have added box_border_search function.
Neighbors_touching_cells.ipynb - This program is to find the number of neighbors touching the object.
Polygonality Hexagonality.ipynb - This program is calculate the polygonality score , hexagonality score and hexagonality standard deviation.

@Priyaaxle @Nicholas-Schaub may I chime in here with the recommendation of Cookiecutter and this template: https://github.com/audreyr/cookiecutter-pypackage
Using that you can very easily setup everything, including setup.py with reasonable defaults.
Hope it helps.

@Priyaaxle @Nicholas-Schaub may I chime in here with the recommendation of Cookiecutter and this template: https://github.com/audreyr/cookiecutter-pypackage
Using that you can very easily setup everything, including setup.py with reasonable defaults.
Hope it helps.

I have been meaning to set up a cookie cutter for a Python WIPP plugin and Python utility, but haven't. This is an excellent suggestion. Maybe we should make this a task for one of the data scientists.

I am closing this for now since there will be radical changes to this plugin.
To summarize our offline discussions, we settled on turning this into a containerized, command line function. The only thing the code will do is get features developed by @Priyaaxle or already present in scipy.


Incredibly slow read speeds!

Heyo, I used this hashmap implementation for my small programming language and it works pretty well, however, the read speeds are prolonged compared to other implementations and write speeds of this one. Is that a bug? It may be a good idea to make some benchmarks.
If speed is not the primary concern of this lib but rather the use of UTF8 encoded strings, then I'm sorry to trouble you with this issue and you can ignore it.
Best regards!

I don't quite understand how the UTF8 encoded strings comes into it - but I agree we could do with benchmarks on the library! hashmap.h was more about the ease of functionality, rather than speed. There are much better hashmaps out there, but most do not support C.

There are much better hashmaps out there, but most do not support C.

There's hashmap.c at https://github.com/tidwall/hashmap.c
It seems to be faster when it comes to reading data, but yeah, I like the ease of your implementation more. Keep up the good work


Ability to launch static hosts in an aws placement group.

Launch hosts in a given aws placement group

Terminate all hosts in a teletraan group

@lilida

r+