Download error

Version 1.7.16, test url - "https://youtu.be/XTV13SDWB-g"
problem description: the video will not download and an error message will be displayed.

I'm having the same error here.
I think YouTube changed something in the API

HI, this is a problem with the library that the program uses. Wait until the next version is launched and sorry.

HI, this is a problem with the library that the program uses. Wait until the next version is launched and sorry.

It downloads with YoutubeExplode 6.0.0-alpha2

Yes, i made my own version of the program with that version. I needed to change some parts of the code, but no problem.

Working just fine for me
https://ibb.co/XyR3drb
I use the demo method, it's something you did.

Maybe u are using this version of the library: YoutubeExplode 6.0.0-alpha2. With that version, the downloader works fine. i think im goingo to make a pull request lol

I'm using sharpgrabber... just sayin

XDDD

Yes, i made my own version of the program with that version. I needed to change some parts of the code, but no problem.

Can you share your own version? Thank you.

Yes, i made my own version of the program with that version. I needed to change some parts of the code, but no problem.

Can you share your own version? Thank you.

Well, it's in Spanish so let me do a version in English

or i can share u the spanish version, but u will need to follow the UI or translate all.

https://mega.nz/file/v4gSzKqS#hUPwxBJe5wk9PF2CP85HWPOCWMDAnGAcGTqjk0N187o

https://mega.nz/file/v4gSzKqS#hUPwxBJe5wk9PF2CP85HWPOCWMDAnGAcGTqjk0N187o

There is only 2 files inside, it's supposed to be like that?

or i can share u the spanish version, but u will need to follow the UI or translate all.

It's working, thanks a lot!

Thank you for the fix but for some reason the interface is now in Spanish?

Thank you for the fix but for some reason the interface is now in Spanish?

Because this version that i made is for getting update by me for my friends and we are Spanish.

Give me 8 hours that i come back from school and i'll do the fix but in English.

Thank you for the fix but for some reason the interface is now in Spanish?

Checkout my PR and compile it yourself or download the Workflow Build files, it's in English.

Thank you for the fix but for some reason the interface is now in Spanish?

Checkout my PR and compile it yourself or download the Workflow Build files, it's in English.

https://github.com/Tyrrrz/YoutubeDownloader/actions/runs/730271308
The Link to the PR

Thank you, works perfectly and it's in English! 👍🏻


Add virtual rendering to timeline rows when scrolling

For timelines with potentially many rows, rendering rows that are out of view is an inefficient use of browser resources. This ticket is to implement a virtual rendering scheme when scrolling that only adds rows to the DOM that are currently in view for a timeline.

Here is an existing virtual list component we could look at using: https://github.com/sveltejs/svelte-virtual-list

Could also look into using something like: https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API


haskellPackages.sensei: mark unbroken, override

Description of changes

Sensei had an issue on NixOS relating to tmpfs, and 0.6.2 fixes that.
But it still needs an override because it (the binary and the tests)
require the git binary and latest hspec.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Also, please feel free to add yourself as a maintainer for this package in pkgs/development/haskell-modules/configuration-hackage2nix/main.yaml.


Release: v0.15.0 (Ropsten)

Bumping tbtc to fresh rc and bringing in the latest keep-ecdsa rc.

All done here, closing.


Add a semi-automatic test for #221

This needs to be given a valid font as a command-line argument, but is
otherwise automatic.

Includes #223.
I haven't wired this up to the CMake build system or the CI, because I don't want to put this on the critical path for 2.20.0, but it does seem to work.

Given how long it took to get the SDL_image test passing reliably, if @slouken is doing 2.20.0 soon then I'm inclined to wait for 2.21.x to add this - it's just not as high a priority as the CMake build system or getting a release out.

Given how long it took to get the SDL_image test passing reliably, if @slouken is doing 2.20.0 soon then I'm inclined to wait for 2.21.x to add this - it's just not as high a priority as the CMake build system or getting a release out.

I'm okay with adding this now. The SDL_image test found bugs that were good to fix for the next release, so if that happens here, that makes me happy. :)

OK, I'll try to get it integrated into CI in a way that isn't randomly regressing, but please don't block on this for 2.20.
(Also please don't imagine that this is a thorough test - it doesn't even render any text.)

@madebr, I'm working on updating SDL_ttf 3.0, do you want to take a look at this again?


[WIP] download page

Created a dedicated download page for multi-version release

@chenliang613 please clone and check download page. Added a dedicated page to download the targets directly

@chenliang613 please review and merge


[1.13.x] Add support for dns and dns_search in deploy using composefile

Related to #29685.
The changes is small but this would need a change in anaand/compose-file and needs vendoring. If we are ok to do this in 1.13.x I will make a PR in anaand/compose-file and update this PR. I will also make a PR against master afterwards.
/cc @thaJeztah @dnephin @icecrime
Signed-off-by: Vincent Demeester<EMAIL_ADDRESS>

design LGTM, but not sure we can still take this for 1.13.0

I don't think we should make this change for 1.13

Sounds fair 👼
I'll close that and add support on master a little bit later 😝

I created https://github.com/docker/docker/pull/32059 for the same bug. Would be nice to get this bug fixed soon.


Classloader Question

Hi CookieMonster,
I've recently watched your presentation from the PowerShell Summit 2017.
You mentioned it wouldn't be a great idea to use the ScriptsToProcess for loading classes from modules into the parent scope. Unfortunately there isn't any goo documentation or best practice therefore and I couldn't find any better solution.
So how did you solved this problem? It would be great to get some advice from such a pro like you ;)
If you are interested in the exact problem your can take a look at my repo called Ponduit.

Hi!
I've seen a good number of folks either:

Include all necessary classes in the psm1
Keep classes in separate files, during build process, concatenate them into psm1 as needed (e.g. PoshBot)

Might be other options as well - if you drop by the PowerShell Slack team (invite @ slack.poshcode.org), you can ping folks like Jaykul and others who are more involved in writing modules that use classes
Cheers!

Closing this out - if you end up using any interesting approaches feel free to reply with them - cheers!


feat: database migration

Codecov Report
Attention: Patch coverage is 60.00000% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 87.21%. Comparing base (212e61c) to head (2748054).

Files
Patch %
Lines

src/store/mod.rs
50.00%
6 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #106      +/-   ##
==========================================
- Coverage   87.68%   87.21%   -0.48%
==========================================
  Files          14       15       +1
  Lines         528      532       +4
==========================================
+ Hits          463      464       +1
- Misses         65       68       +3

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Automatically convert tag formats between sources

Some sources use different tag format's, for example some_tag another_tag, Some Tag + Another Tag, Some_tag && Another_tag, etc.
The program be able to make a single multi-site search by automatically converting the case, work separators and tag separators. Also, the user should be able to tell which is his "favorite" format that the program should use to convert searches from.
:warning: It must be possible to disable it on demand or globally for some more complex queries.

Also may be connected with https://github.com/Bionus/imgbrd-grabber/issues/916

Yes, related, but that one is for display of tags while here is for search only (so it also includes separators).
Standardization applies to both though. :+1:


alert prompt dialog input username and password

The problem
I have a website that prompt username and password signin before landing on the website. May I know how to input username and password to it and click ok button.
Details
I have a website that prompt username and password signin before landing on the website. May I know how to input username and password to it and click ok button.
Tell us about your environment

Selenide Version: 6.8.1
Chrome\Firefox\IE Version: 110.0
Browser Driver Version: -
Selenium Version: 4
OS Version: Win 10

Code To Reproduce Issue [ Good To Have ]
I did have alert.accept() and alert.dismiss().

@nicholaswkc34 What type of dialog you mean? There may be many possible options.
If you mean Basic Auth, then Selenide has special methods:
  open("http://localhost:8080/basic-auth/hello", "", "scott", "tiger");


NameError: name 'OrderedYaml' is not defined

May I ask if anyone has encountered this problem?
Traceback (most recent call last):
File "E:\Pythonfiles\image-restoration-sde-main\codes\config\deraining\train.py", line 16, in
import config.deraining.options as option
File "E:\Pythonfiles\image-restoration-sde-main\codes\config\deraining\options.py", line 15, in
Loader, Dumper = OrderedYaml()
NameError: name 'OrderedYaml' is not defined

Hi, you may need to adjustment the line 10 and 11 in options.py to make it work on windows. As suggested here: https://github.com/Algolzw/image-restoration-sde/issues/6#issuecomment-1529268012


Logging does not successfully use hec_ca_file for cert verification by default

When running splunk-kubernetes-logging 1.1.0 on OpenShift 3.11, I encountered the following error:
[warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2019-04-11 23:31:24 +0000 chunk="234234234234" error_class=OpenSSL:SSL::SSLError error="SSL_connect returned=1 errorno=0 state=error: certificate verify failed (unable to get local issuer certificate)"

In order to work-around the problem, I added the following environment variable definition to the Logging fluentd-hec container in the Logging Daemonset:
env:
- name: SSL_CERT_FILE
  value: /fluentd/etc/splunk/hec_ca_file

Thank you. i have added this environment variable as another layer of ensuring ssl verification for openshift.

Hi Rock,
Adding this variable works fine for when you provide a custom certificate chain in your values.yaml (splunk.hec.caFile), but it will break the agent functionality for any installations where the Splunk server uses a certificate issued by a trusted certificate authority and the chain can be verified with the default OS CA file (/etc/pki/tls/certs/ca-bundle.crt).
What happens is that when don't provide a CA certificate file (splunk.hec.caFile), the file /fluentd/etc/splunk/hec_ca_file doesn't get created.  This results in agent's inability to verify the chain of trust of a properly issued certificate.
One can work around it by making the splunk.hec.caFile field mandatory and force people to populate it with certificate chain already present within the Docker image (at /etc/pki/tls/certs/ca-bundle.crt), but it's going to be an ugly hack.
You could extract the value of SSL_CERT_FILE  and set it according to an entry in the values.yaml (ex. splunk.hec.caFilePath) manifest, but I think it would be better to just set it to "/fluentd/etc/splunk/hec_ca_file" only if splunk.hec.caFile is set in values.yaml.
...or in pseudo-code:
if splunk.hec.caFile present in values.yaml
then
  set SSL_CERT_FILE=/fluentd/etc/splunk/hec_ca_file
else
  don't set SSL_CERT_FILE variable

If not possible to dynamically change the daemonset/deployment to leave out the SSL_CERT_FILE variable out entirely, the following would work just as well until the base image changes to one with different path to the OS-provided CA file:
if splunk.hec.caFile present in values.yaml
then
  set SSL_CERT_FILE=/fluentd/etc/splunk/hec_ca_file
else
  set SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt

Thank you for detailed explanation. we will review and see what would be the best way to handle it.

We are seeing issues related to this as well.

with 1.4.6, you can now customize env var for logging daemonset.

Hi @rockb1017
Thank you for the quick turnaround.  There are a couple of issues with this implementation though.

If you leave the environment variable SSL_CERT_FILE out, as suggested by the comment in the values.yaml, it defaults to /fluentd/etc/splunk/hec_ca_file in the resulting daemonset manifest instead of leaving the variable out and defaulting to the one provided by the OS.  I can still set SSL_CERT_FILE to what I need instead of relying on the default and that works for me, but the comment is misleading.  You might want to fix the comment or change the behavior to match it.
The fix is good only for splunk-kubernetes-logging, but we need it for splunk-kubernetes-objects and probably splunk-kubernetes-metrics.

Thanks,
Stoyan

Thank you for the feedback. I will edit the comment and also include this for other pods for 1.4.7!

 Set SSL_CERT_FILE environment variable appropriately when using OS CA file
is this good?

How about:
CA cert bundle location.  Default is "/fluentd/etc/splunk/hec_ca_file".

Personally I think that the default behavior when this variable is not present in the values.yaml file should be that it's not present in the respective daemonset or deployment as well.  This will lead to the daemon using the OS provided CA certs, which is more of an expected behavior than the OpenShift specific "/fluentd/etc/splunk/hec_ca_file".  OpenShift and anyone else's requirements can be accommodated by setting this variable to a custom value, which "/fluentd/etc/splunk/hec_ca_file" really is.  Defaulting to a specific use case doesn't seem like a good idea.  OpenShift is just one of plethora of Kubernetes implementations and is no more special than EKS, GKE, or vanilla Kubernetes installation.
Thanks,
Stoyan

@rockb1017  as I wrote about in the PR, unsetting doesn't work. ;-) https://github.com/splunk/splunk-connect-for-kubernetes/pull/560#discussion_r578707247

Set SSL_CERT_FILE environment variable appropriately when using OS CA file

What is the "appropriate OS CA file"? the OS of the K8s platform is maybe not the same OS as used in the container and paths might not be the same. So a user could set it to the k8s platform OS path and then it doesnt work too. leaving room for new issues about TLS not working.

the OpenShift specific "/fluentd/etc/splunk/hec_ca_file"

@sstoyanovucsd  hmm... OpenShift doesn't care about your TLS config in containers. Can you explain what you mean with that this setting is specific to OpenShift? I do not see that it is related to OpenShift.

Personally I think that the default behavior when this variable is not present in the values.yaml

@rockb1017 I second that. But it would be a breaking change for existing users. Maybe it should only be configured if a user has set the caFile option?

Well noted. Thanks @vinzent @sstoyanovucsd !
I have made changes here.
https://github.com/splunk/splunk-connect-for-kubernetes/pull/576

Thank you @rockb1017.  I'll test as soon as the changes make their way into a release and let you know how it goes.

Great timing I just found this issue this morning trying to install splunk connect on my EKS cluster.
Any idea when the release will be available?

i am waiting for internal approval for next release. hopefully next week?  ;)

@rockb1017 I'll wait for the fix in the next 1.4.7 version. Have the same issue here.
Does anybody know when the next version will be released?

it is currently assigned and being reviewed. Thank you for your patience!

@rockb1017, I just tested version 1.4.7 and it works as expected.  Thank you for your time and effort you put into this project!

@sstoyanovucsd Thank you for confirming!


mysql connections lost; treated as fatal

reproducable in v5.2.1
I start an MySQL connection with transaction support. The server is MariaDB and the sqlpad driver in mysql
run a query.
It seems that after a predefined time the session/ connection is closed automatically. (In my case it seems to be after 2 minutes)
   2020-07-28T11:33:42.03+0300 [APP/PROC/WEB/0] OUT {"level":50,"time":"2020-07-28T08:33:42.035Z","pid":7,"hostname":"ed954926-3567-4772-455c-19fc","name":"sqlpad-app","fatal":true,"code":
"PROTOCOL_CONNECTION_LOST","stack":"Error: Connection lost: The server closed the connection.\n    at Protocol.end (/usr/app/node_modules/mysql/lib/protocol/Protocol.js:112:13)\n    at TLSSocket.onend
(_stream_readable.js:679:10)\n    at Object.onceWrapper (events.js:417:28)\n    at TLSSocket.emit (events.js:323:22)\n    at endReadableNT (_stream_readable.js:1204:12)\n    at processTicksAndRejections
(internal/process/task_queues.js:84:21)","type":"Error","msg":"Connection lost: The server closed the connection."}

moreover, trying another query, I get (on the GUI) Cannot enqueue Query after fatal error. The only way to get rid of this is to reselect the conenction from the connection list or refresh the screen.
I would expect the connection to be closed so soon.
I would definitely expect  a new connection to be created in this case.
I think I would like some visual feedback for "new connection created". Maybe related to #806 - the client can remember which connection was used. If it was dismissed by user action - drop response silently. If the connection was replaced because of server side decision, display a notification somewhere.

mysql2 seems to do the same, but a different error message is displayed

another workaround is to disconnect + re-connect using this icon:

I think what needs to happen here is the mysql databse connection needs to be used continuously to prevent it from closing?
For example maybe a SELECT 1 is sent every min or so to tell MySQL not to close the connection.

I've seen somewhere that js sql driver supports ping() for keep alive.
I wonder if what the mysql cli do to avoid being disconnected.
This keepalive is important if one wants to experiment with transactions.

This suggests that there can be different timeouts for interactive vs non interactive connections. Sqlpad connections should be flagged as interactive somehow.
https://stackoverflow.com/questions/51506416/mariadb-server-times-out-client-connection-after-600-seconds

@eladeyal-intel are we okay to close this now?

I now realize this is a duplicate of #699. Let me test this and update.


return 503 Service Unavailable instead on 500 Internal Server Error

Expected Behavior / New Feature
It would be nice if Ocelot returned 503 when Ocelot cannot connect to the downstream service, e.g. when a connection is refused.
When running in docker or other container environments, a service can be stopped temporarily due to updates or other things. In such cases 500 might be misleading.

Our current errors logic: Server error responses
But we have some plans to forward original status code. So, overriding status code now is very bad.
See #926
See all knowledge domain: "status code"
But you can solve the 500 problem by Delegating Handlers.
Override status code for your needs in custom delegating handler.


additional_datasets - Obs4MIPS day of year failure

Hi,
I'm evaluating the CMIP5 sea surface temperature field, tos, using the Obs4MIPs dataset, tos_ATSR_L3_ARC-v1.1.1_199701-201112.nc. This dataset can be obtained here.
For an unknown reason, the preprocessor fails to calculate the day_of_year value and instead sets all values to 1. Ie:
In [2]: nc01 = ncdfView('01_fix_metadata.nc')
In [3]: nc01('day_of_year')
Out[3]:
array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])

This occurs in the early stages of the preprocessor, between 00_load_cubes.nc (which does not include the day_of_year field) and 01_fix_metadata.nc, where it is broken.
I can assume that might be fixable by setting a fix_metadata function for the additional dataset. How do I do that?
Thanks,
Lee

Hi,
This bug is still present. Is there any chance that it will be fixed soon-ish?
If not, @valeriupredoi  and I may have to show this kind of rubbish to the Crescendo project Community at the meeting next Monday:

Cheers,
Lee

Yes, a fix function it's the right way to go.
Basically you need to name the fix file as the dataset you want to fix and put it in a subdir for the corresponding project (obs4mips in this case). Then you can just adapt one of the existing fix file to your case.

yup - see eg https://github.com/ESMValGroup/ESMValTool/blob/version2_development/esmvaltool/cmor/_fixes/OBS/BDBP.py only you will need to create obs4mips instead of OBS

out of curiosity, why is the day of year important if you are performing monthly analyses?

I'm not using it directly, but iris.quickplot.plot() determines the values along the x axis of these plots using year and day of year, I think.

aha so then what does the day have to do with those strange delta-function-like lines? why would it matter if day_of_year = 1 or any other day if the month is correct? I feel that we may introduce a fix that's not really needed here...

Have you try to compare with other datasets, to check whether day_of_year is defined in a different way? I also suspect, like @valeriupredoi, that for monthly means this variable does not really make sense.

@ledm if you introduce the fix in /cmor/_fixes/ this will get it applied to the ATSR files no matter what type of analyses you do, so for a daily analysis this fix will destroy your data; also, try as much as possible not to rely on iris.quickplots for simple plots like this one, in fact, the iris quickplot is really not well built and should be used only when we can't use matplotlib out of the box; in _multimodel for instance all days are set to the 1st of the month for a monthly mean analysis in an ad-hoc manner

I've been looking at this further, and I think that the problem isn't that the day_of_year aux_coordinate field is faulty, but rather that the num2date function doesn't correctly calculate the day of the year in the additional datasets.
ie:
time_coord = cube.coord('time')
times = time_coord.units.num2date(time_coord.points)
print(times[0])
  cftime._cftime.DatetimeJulian(1997, 1, 16, 0, 0, 0, 0, -1, 1)

The final digit there is the day_of_year, the order is  (year, month, day, hour, minute, second, microsecond, day of week, day of year.
Where as the same set of commands on a model output produces:
cftime._cftime.DatetimeNoLeap(1960, 1, 16, 12, 0, 0, 0, 3, 16)

It's as if the cube.coord('time').units.num2date doesn't work with the  Julian calendar using the cftime._cftime.DatetimeJulian datetime.

Yes, that's the problem, a bug in cftime
import cftime
In [6]: cftime.num2date(16,'day since 1950-01-01 00:00:00.0000000',calendar='365_day')
Out[6]: cftime._cftime.DatetimeNoLeap(1950, 1, 17, 0, 0, 0, 0, 1, 17)

In [7]: cftime.num2date(16,'day since 1950-01-01 00:00:00.0000000',calendar='julian')
Out[7]: cftime._cftime.DatetimeJulian(1950, 1, 17, 0, 0, 0, 0, -1, 1)

In [8]: cftime.num2date(16,'day since 1950-01-01 00:00:00.0000000',calendar='360_day')
Out[8]: cftime._cftime.Datetime360Day(1950, 1, 17, 0, 0, 0, 0, -1, 17)

The cftime Julian calendar is buggy.

I think the solution may be to convert to a more sensible calendar. (Seriously, we stopped using the Julian calendar in 1582! Who thought it would be a good idea to use it for climate science data?)
Is there an easy way to convert calendars and time units using iris?

bahahah, we missed you, @bjoernbroetz :grin: yes, one can use convert_units: https://scitools.org.uk/iris/docs/latest/iris/iris/cube.html#iris.cube.Cube.convert_units but probably the best and most secure way is to use cf_units and convert first to a date object then use date2num with the correct calendar @jvegasbsc may have better ideas since he's the Timekeeper in esmvaltool :grin

Nope, it's not possible to do it with convert_units.
        time_coord = cube.coord('time')
        newUnit = cf_units.Unit('day since 1950-01-01 00:00:00.0',calendar='gregorian')
        time_coord.convert_units(newUnit)

produces:
ValueError: Unable to convert from 'Unit('day since 1950-01-01 00:00:00.0000000', calendar='julian')' to 'Unit('day since 1950-01-01 00:00:00.0', calendar='gregorian')'.

ah yes, because stoopid calendars are different @ledm look at the julian functions here: https://scitools.org.uk/cf-units/docs/latest/utilities.html cf_units.date2julian_day(date, calendar) or the one above it should do the trick for you

Related to https://github.com/Unidata/cftime/issues/13 and https://github.com/Unidata/cftime/issues/70

I reported this issue of the unidate/cftime github page here: https://github.com/Unidata/cftime/issues/70#event-1890824922
The cftime developers said that this issue was fixed in the cftime master and will be in the next release. (Currently they're on 1.0.1, but the fix is not in that one.)
Do we work with the up to date version, or do we need to update the environment.yml file?

If  you do a new development install of esmvaltool you get version 1.0.1 at the moment, to see this kind of information you can go to circleci: https://circleci.com/gh/ESMValGroup/ESMValTool/
Click on the development branch, select a develop or other test run and click artifacts, there we keep all versions of software used, see e.g.:
https://circleci.com/gh/ESMValGroup/ESMValTool/7105#artifacts/containers/0
and click on requirements.txt

The fix is not in v1.0.1, it's in the cf_units master, which doesn't have a version number. How do I get the latest master version into a development branch of esmvaltool?

Either wait for a release or clone the repository and do an installation from the repository

@ledm can you check again it there is a new release solving your problem?

#832 will automatically update cf_units to 2.0.1 out the box, so may test after that PR gets merged

the time aligner function from this does it now, cf units 2 and all https://github.com/ESMValGroup/ESMValTool/pull/867

@valeriupredoi can we close this issue?

cf_units is at 2.0.1 or .2 (depending which conda channel is used ie
cf20190130 has 2.0.1) so I think we just need to check if the fix is there
(and it should be) and we can then close the issue. Didnt bring my laptop
to EGU, so can someone check pls :beer:
Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom
On Wed, 10 Apr 2019, 11:04 Mattia Righi<EMAIL_ADDRESS>wrote:

@valeriupredoi https://github.com/valeriupredoi can we close this issue?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ESMValGroup/ESMValTool/issues/603#issuecomment-481606416,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbpCo18YtzgAkK_MQm1wIrGyHrsYapt5ks5vfakEgaJpZM4Wi-b1
.


Port over an updated WindowController.js

There is an updated version of WindowController.js that doesn't rely on magic numbers for timing between tabs.
https://github.com/fastmail/overture/blob/master/source/application/WindowController.js
However it relies on code that's internal to this Overture library. We should try to port over this code and use just pure JS so we can update WindowController.
I don't think this is absolutely necessary but the code looks cleaner.

This could be related to this: https://github.com/CTFd/core-beta/issues/35


Console spam

I'm trying to figure out some console spam I got after I put in your mod and made several composter recipes. Doesn't seem to affect anything, just fills the console - the new composter stuff works just fine and the console warnings don't seem to happen when I'm using a composter or the new recipes, just occasionally it happens a few times or I get a wall of it.
I was hoping you'd seen something like this before and would know if I screwed something up
12.06 19:59:50 [Server] Server thread/WARN Ignoring unknown attribute 'generic.reach-entity-attributes.reach'
12.06 19:59:50 [Server] Server thread/WARN Ignoring unknown attribute 'generic.reach-entity-attributes.attackRange'
Server log:
https://pastebin.com/hiTku5h7
Client log:
https://pastebin.com/vZQTus19

actually I think it's from removing another mod. Those are the only two things I did, and it seems far more likely the other mod had affected entity anything :)


N_OUTSTANDING

N_OUTSTANDING is a parameter when using cmake to generate project, but in hw/hdl/network/rdma/rdma_flow.sv, the RDMA_N_OST is hardcoded as 16. Is this a bug or feature?

It should be a variable. @maximilianheer can you look into this?


Displaying generic image collections

🚀 Feature
Napari currently assumes that within a viewer, all images occupy the same physical space. This assumption is invalid for a lot of collections of images, e.g. sample images in skimage.data:
import skimage.data as skid
list_of_images = [skid.astronaut(), skid.clock(), skid.brick(), skid.binary_blobs(length=50,n_dim=3)]
# This would be intuitive, but it doesn't work -- array dimensions don't align
# napari.view_image(list_of_images)

# This works, but the logic of the viewer is weird -- the images are layered, seeing them individually is hard
v = napari.viewer.Viewer()
[v.add_image(d) for d in list_of_images]

This renders all the images from list_of_images in the same coordinate space, and gives them all a Z axis slider even though most images are 2D. Seeing these images individually thus requires extra effort (layers above and below must be masked), and there is are vestigial GUI element (the z-slider, and the "roll displayed axes" button).
Ideally (from my perspective) calling napari.view_image(list_of_images) would generate a single image layer with a slider controlling which image is rendered, and additional sliders appearing as needed per image (e.g., the blobs image is 3d, so it needs an extra dimensions slider).
Motivation
For my work, I need to view a large collection of 3D/4D images that have different dimensions and are from totally different spaces, so it doesn't make sense for them to be overlayed, and it doesn't make sense for them to share dimension sliders. But I would like to quickly browse them!
These images are human-annotated training data for a machine learning application; I suspect that lots of training data for 2D/3D ML will be collections of arrays with different sizes from different spaces, so the issue I'm facing might be fairly general.
Pitch
A generalized image viewer for collections of independent images. This could be accomplished via an internal flag that determines whether the viewer should render images in the same space or not; this flag could be set based on whether a) the user supplies a kwarg, e.g. napari.view_image(data_3d, shared_coordinates=False) b) whether the user supplies a collection of images with different sizes /shapes, c) some other heuristic.
Additional context
This problem reminds me of the perennial "is this last axis RGB, or is it just an axis that happens to have length== 3" problem. I think the solution there is to assume the general case, and allow the user to supply flags for the special case. The same approach might work here, once we figure out which case is specific and which is general. From an abstract perspective, the NDArray is a special case of collection of arrays, but empirically I'm not sure what which type of data people are viewing more often.

This also relates to #605. I need to think about it more, but it would be pretty straight forward now if you can package everything inside one large dask array, as dask image imread does. If the biggest dim image have dim 5 you'd get 3 sliders, but it should still be fine.
Can you try packaging list_of_images inside a dask array?

putting list_of_images in a dask array requires first putting it in a numpy array, and this still generates errors when passed to napari.view_image. The first error I got was in calculating the contrast limits, and the second error was in interpolation during construction of the thumbnail image.

@d-v-b I think #605 captures this quite well. Having said that, using layers, then switching to grid mode is probably pretty effective as a "for-now" workaround?
I don't know what you think is the correct behaviour for mixed 2D and 3D images, though. I think it would look pretty terrible to have one of the sliders popping in and out of existence...?

@d-v-b I think #605 captures this quite well. Having said that, using layers, then switching to grid mode is probably pretty effective as a "for-now" workaround?

Yeah, at the moment I can work around the issue. In the worst case I could define a dask array that does padding to the maximal dimensions, thereby ensuring that all axes are identical.

I don't know what you think is the correct behaviour for mixed 2D and 3D images, though. I think it would look pretty terrible to have one of the sliders popping in and out of existence...?

I agree that "surprise sliders" should be avoided, but it's also bad to have a slider that doesn't do anything for the current image. What if sliders were enabled/disabled based on the dimensionality of the current active layer? The number of sliders would still be determined by the maximal dimensionality across layers and (thus constant while viewing a fixed set of layers), but sliders would get greyed out if you select a 2D image layer. Might be worth trying?

The dask array padding works quite nicely when they all have the same dims - and this is certainly something we could consider doing under the hood if it was desirable - people could pass a list of images and we'd take care of the rest, for example:
from skimage import data
import dask.array as da
import numpy as np
import napari

list_of_images = [data.moon(), data.clock(), data.coins()]
max_shape = np.max([im.shape for im in list_of_images], axis=0)
full_array = da.concatenate([da.pad(da.from_array([im]),
                                    [(0, 0)] + [(i//2, (i+1)//2)for i in np.subtract(max_shape, im.shape)],
                                     mode='constant') for im in list_of_images])
napari.view_image(full_array)

We could even do this for cases where the dims weren't the same if really wanted. I'd probably put this intercept in the viewer.add_image function so that we didn't change the Image model, but just did this conversion there. It would also mean we could move towards supporting #605 more easily.

I'm not sure I like the idea of napari padding all images in a layer to make them the same size... I would not expect that an image viewer would add data to my images. E.g., padding leads to napari erroneously reporting that the outside of the image has value 0. And when we start to think about arrays with differing dimensionality, the memory cost of padding a 2D array up to 5D might be prohibitive.
More generally, I think it would be better to avoid coercing input data with diverse size / shape into an NDarray. The axes in the skimage sample images are not the same, so putting these images in an NDarray via padding just seems weird. To me, the natural data structure for these sample images is just a 1D collection, e.g. a list. If the image_layer is committed to only supporting NDarrays, maybe this use case is just out-of-scope for that layer?

If the image_layer is committed to only supporting NDarrays, maybe this use case is just out-of-scope for that layer?

I wouldn't say committed to only, but that is the primary use case, and I am reluctant to overly complicate the code to support secondary use cases. If we can come up with a simple, transparent solution that works, I'm all for it. Part of the reason that it really complicates things is that now if you switch to 3D, you must instantiate a "cube" of data. Or if you roll the axes — what happens then? Maybe we need a different kind of interaction here.

This problem surfaced again for me -- I wanted to view the output of a bunch of different filters applied to the same data (pursuant to playing with this example of the ilastik-style functionality within napari). I.e.:
im_gauss = skimage.filters.gaussian(image,sigma = 10, preserve_range = True)
im_gauss2 = skimage.filters.gaussian(image,sigma = 20, preserve_range = True)
im_frangi = skimage.filters.frangi(image)
im_prewitt = skimage.filters.prewitt(image)
im_meijering = skimage.filters.meijering(image)
im_gauss_laplace = skimage.filters.laplace(skimage.filters.gaussian(image,sigma = 5,preserve_range=True),ksize=10)
im_gradient = skimage.filters.rank.gradient(image,skimage.morphology.disk(8))
im_entropy = skimage.filters.rank.entropy(image,skimage.morphology.disk(8))
im_roberts = skimage.filters.roberts(skimage.filters.gaussian(image,sigma = 5,preserve_range=True))

all_filt = [im_gauss,im_gauss2,im_frangi,im_prewitt,im_meijering,im_gauss_laplace,im_gradient,im_entropy,im_roberts]
names = ['Gauss $\sigma$=10', 'Gauss $\sigma$=20','Frangi','Prewitt','Meijering','Gauss+Laplace', 'Gradient',
        'Entropy','Roberts']

Since all the images in all_filt are the same size, they can be put in a numpy array, which would in principle be nice for viewing with napari. However napari assigns a single LUT to the entire array, so no filtered image is visible -- the dynamic range of the gaussian-filtered images is totally different from the dynamic range of the edge-filtered images.
The alternative is to make a single layer per image, but then to view a single image, you have to hide all the images on top, which can amount to ~O(num_images) clicking.
I think this use case, and the one I have detailed above, would not be too hard to accommodate: add an add_image_collection() method to the Viewer that takes a 1D collection of images. Adding this layer to the viewer creates a slider that indexes the the collection. If the collection contains 3D or higher images, additional sliders are created, but these sliders are disabled when they do not match the dimensionality of the currently displayed image. Likewise for the rollaxes and 3D viewer buttons: these would only be active when the currently displayed image from the collection supports them.
This image_collection layer would move some of the logic of multiple image layers to a single layer. This would be fine on its own, but I don't think it would work with the current napari model where all the layers are in the same (implictly defined) viewer space. So I think sorting out https://github.com/napari/napari/issues/659 is a prerequisite for sorting out this issue...

@d-v-b indeed, I do think need to figure out our real-space interaction before tackling image collections.
I wonder if this specific use case, though, would be solved by a more powerful layer interaction model. These images all do refer to the same space, after all, so I think layers is the right conceptual model. People working with multispectral data have noticed that napari begins to fray at the edges with too many layers, so it’s something that needs improvement in various scenarios.


Merge google -> main

6ad66c5d Merge pull request #2792 from hanhanW:main-to-google
c7b38af5 Synchronize submodules
4959014b Integrate TF at tensorflow/tensorflow@c5543bc170cc
2a9b208f Synchronize submodules
5c71d7bc Integrate LLVM at llvm/llvm-project@acb66b9111ba
1c9b8367 Clean up repo alias list

Oops...forgot to check commit summary, seems that the extension didn't work right. :(

Yeah it's a bit buggy; GitHub is doing something weird things to the event listeners (https://github.com/zachwhaley/squashed-merge-message/issues/11). At least it's a merge commit and doesn't have any Copybara tags in it :-D

yeah, that's indeed a good news, haha!


Not all codes are present

MIssing code AADB2C99002

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: fd9c64ff-5065-9a2b-4965-c48685482fe1
Version Independent ID: 3ff9871f-d28d-b752-750e-1142556ec8cb
Content: Error code reference - Azure AD B2C
Content Source: articles/active-directory-b2c/error-codes.md
Service: active-directory
Sub-service: b2c
GitHub Login: @kengaderdus
Microsoft Alias: kengaderdus

@adduss
Thanks for your feedback! We will investigate and update as appropriate.

Hi @adduss
It seems like you have set up Azure AD B2C in company B's tenant to authenticate users from company A's Azure AD. The error "AADB2C99002: User does not exist. Please sign up before you can sign in" occurs when the user is not found in the Azure AD B2C directory.
To resolve this issue, you need to configure Azure AD B2C to use company A's Azure AD as an identity provider. This way, users from company A can authenticate using their Azure AD credentials, and Azure AD B2C will not look for them in company B's directory. Here are the steps to configure Azure AD as an identity provider in Azure AD B2C:
Register an application in company A's Azure AD tenant.
Record the Application (client) ID and create a client secret for the registered application.
In company B's Azure AD B2C tenant, create a policy key to store the client secret.
Configure Azure AD B2C to use company A's Azure AD as an identity provider by adding it to the ClaimsProvider element in the extension file of your policy. After configuring Azure AD as an identity provider, you should be able to authenticate users from company A in the Web App deployed in Azure from company B using Azure AD B2C.
We are going to close this thread as resolved at this moment once the document's author approves, we'll update it as necessary changes. but if there are any further questions regarding the documentation, please tag me in your reply and we will be happy to continue the conversation.


FileWriter with append enabled is not working as expected

Describe the bug
Writing files using the filewriter module with append enabled is not working as expected. Multiple files are getting created instead of one single file.
To Reproduce
Steps to reproduce the behavior:

Write any file using filewriter with append property enabled.
When the output path is a pattern, it will generate multiple files rather appending to one file. Also creates an empty dummy file.
When the output path is not a pattern, an error "output path is invalid" error is raised but the file gets written.

Expected behavior
In both cases, expectation is to get one file as the only output file on disk with all the content.
Desktop (please complete the following information):

OS: Windows 10
Architecture: x64

Would you like to work on the issue?

Assigned to @Vinayak-YB


Question about the option "use_smpl_param_regressor"

Thank you for your awesome research!
What is the difference whether using the option "use_smpl_param_regressor"?

Hello,
If --use_smpl_param_regressor option is set to True, then a SMPL parameter regressor would regress SMPL parameters from the output mesh vertices estimated by FastMETRO. Please refer to Experiments.md and Section C of the supplementary material for more details.
Thanks for your interest in our work!!

Please reopen this issue if you need more help regarding this.


add hdfs storage driver

This is for issue #1086 .
This PR build hdfs package as HDFS Driver Factory,
to presents HDFS Storage Driver to distribution.
This allows distribution client to access Apache Hadoop
Distributed File System server. The connection protocal
is WebHDFS REST API, provided by Apache official.
Signed-off-by: xiekeyang<EMAIL_ADDRESS>

Functionality
This PR build hdfs package, to presents HDFS Storage Driver to Distribution.
This allows distribution client to access Apache Hadoop Distributed File System
server. The connection protocal is WebHDFS REST API, provided by Apache official.
Developer had better to use Hadoop server upper than version 2.6.0, which I mention
in hdfs document.
Dependency Project
this package leverages the vladimirvivien/gowfs project
functions. About this I mention on the head of hdfs.go file.
Design
Here I present a new package of hdfs, as HDFS Driver Factory.
the hdfs package presents new driver files:

hdfs.go provides the dtorage driver APIs.

fs.go provides WebHDFS REST API functions.

vladimirvivien/gowfs project provides full WebHDFS REST API functions to
upload and download data stream. I didn't import the package totally, but leverages
and modifies some necessary functions and structures.
Test
The implementation passes on storage driver unit test and uploading and downloading
actual images, and the performance looks OK. Hadoop client and server are on same host,
and the upload and download speeds is similar with local file system mode.

@xiekeyang I don't have time to review this now, but I would suggest that this driver be called hdfsweb to differentiate from a driver that speaks the actual client protocol. For example, one based on https://github.com/colinmarc/hdfs might be the better candidate the hdfs name.

Looks like we would not be able to merge this PR because of merge conflicts. Please rebase, fix conflicts, and force push to your branch.

@stevvooe
The package name is changed to hdfsweb. About the selection of hdfs protocol as web api or RBC, I present comment on #1086 . I'd like to get your response. Thanks a lot!

@stevvooe  do you have time to take a look on this PR? Thanks a lot!

@xiekeyang I took a quick look over everything. A number of things need to be addressed:

Make sure you always check error conditions.
Make sure that request body lifecycle is appropriately handled. I saw some odd constructions that may lead to leaks. I'd recommend returning a lifecycle object (one that implements io.Closer) if you need to have the Body open passed the execution of the method.
You'll need to add a testsuite runner to circleci. Likely, you'll need to setup and HDFS endpoint to test against. Drivers must be fully tested by the testsuite before being merged.

Getting the above handled should help this move forward.

@stevvooe  Sorry it is my fault that I didn't improve the porting code. I'd check and improve the full code.

@xiekeyang Don't apologize. I am just listing out the items it will take to move this forward.
没关系，你写的挺好。接下来我们就一起努力加油吧。

:smile:

I refactor the hdfs driver, to present as the below,

Every HTTP request is set up according to distribution regularization.

Safe close all HTTP request body.

Handle all errors to return the right and explicit message.

Remove all HDFS APIs unnecessary to storage driver, to make the code clean.

Add the necessary example and description.

Implement unit test code in hdfs_test.go, as factor of storage driver testsuite. I test passed on my local endpoint.

Could you please take a look? Thanks a lot!

@stevvooe
I refactor the hdfs driver, to present as the below,

Every HTTP request is set up according to distribution regularization.

Safe close all HTTP request body.

Handle all errors to return the right and explicit message.

Remove all HDFS APIs unnecessary to storage driver, to make the code clean.

Add the necessary example and description.

Implement unit test code in hdfs_test.go, as factor of storage driver testsuite. I test passed on my local endpoint.

Could you please take a look? Thanks a lot!

Thanks, I have successfully used this backend against my testing HDFS cluster. I hope this patch will get merged soon.

@xiekeyang Is the testsuite being run with circleci?

HDFS testsuite have been run successfully with circleci, using my HDFS cluster.
The HDFS cluster should be set up firstly.
And, please configure the environment parameters like,
export HDFS_NAME_NODE_HOST = <host>
export HDFS_NAME_NODE_PORT = <port>
export HDFS_ROOT_DIRECTORY = <dir>

@stevvooe
The above params is obligatory.
And you can configure the optional ENV parameters like
HDFS_BLOCK_SIZE, HDFS_BUFFER_SIZE and HDFS_REPLICATION.
Finally, exec go test under webhdfs directory (maybe need set timeout).

@xiekeyang I am not seeing any changes to the circle.yml file that would indicate that these tests are running in circle CI. You'll have to setup the HDFS cluster in the circle.yml file and set the environment variables.

@stevvooe OK, understood.

Current coverage is 54.88%

Merging #1123 into master will decrease coverage by -2.67% as of 6c7a403

@@            master   #1123   diff @@
======================================
  Files          116     118     +2
  Stmts        10471   11003   +532
  Branches       719     717     -2
  Methods          0       0
======================================
+ Hit           6027    6039    +12
+ Partial        719     717     -2
- Missed        3725    4247   +522

Review entire Coverage Diff as of 6c7a403

Uncovered Suggestions

+0.37% via ...driver/hdfsweb/fs.go#233...273
+0.35% via ...driver/hdfsweb/fs.go#285...322
+0.33% via ...driver/hdfsweb/fs.go#332...367
See 7 more...

Powered by Codecov. Updated on successful CI builds.

@stevvooe
I add file contrib/hdfs/ci-setup.sh to install hdfs, and add the execution to circle.yml.
Here I set up the hadoop hdfs on host, and configure the hdfs ENV parameters.
The CI and janky test are passed.
Please take a look, thanks so much!

@RichardScothern PTAL

@RichardScothern Thanks so much!

The documentation failure is because your branch is based on an older master, which is missing the meta data for manifest-v2-2.md - it won't cause any issues, but I'm presuming you'll need to rebase for code reasons anyway :)

@RichardScothern  I'm sorry that your review comments lost. But I don't know why. :cry:
So far I only move the replication, buffersize and blocksize parameter to filesystem structure.
But I don't remember others you mentioned.
(likely instead local temp file by io buffer, and present hdfs server internal error message)
Could you please re-paste the comment here?
Thanks very much!

@xiekeyang : My main concern was creating a new http client and transport on each request and the temp file usage for WriteStream

@RichardScothern @aaronlehmann @BrianBland
Appreciate of your review! I read the comments carefully and improve the PR. PTAL

Unmarshal hdfs JSON exception message and report it to upper layer when meeting these kinds of error, and report storagedriver.PathNotFoundError in the case of path not found.
I add the RemoteException structure in types.go as error interface adapter, with description.

Improve WriteStream implementation. I drop creating local temp file way, replacing it by bytes.Buffer.
It read data to the buffer, via ReadFrom. That share the low memory space, and should be a good performance.
Accordingly, remove LocalDirectory parameter and not use TempDir any more.

About HDFS scheme besides HTTP.
We can find in official documentation about webhdfs scheme.
There are 3 schemes, WebHDFS URI, HDFS URI and REST API. REST API is a popular way to access services. So, I select it for this PR. I haven't tried other 2 ways yet. As I know, HDFS URI and WebHDFS URI depend on FileSystem URIs. They need modify some configuration in HDFS server, and need likely implement RPC protocol for RPC port.
As far as I learn it doesn't support HTTPS.

Improve Rename API to avoid request competition by clients.
Rename is not atomic, and it can't over. And HDFS REST API doesn't support copy file from server to server directly.
So I use the for loop way to judge if renaming successfully. If not, then re-delete destination-rename. And if OK, then return.
I use boolean response as rename result. This returns false only when destination  it doesn't raise dead loop risk.
The reference document is [FileSystem rename](http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#rename(org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path))

HTTP transport
I configure the http transport instead DefaultTransport in each http request.
DefaultTransport set dial timeout 30 seconds, which seems to be too long.
s3, swift and some other storage set up new transport. Ex, s3 set dial timeout 2 seconds.
Here HDFS follows s3, to create a new http client and transport on each request.
To improve, we can create client for each request in fs.go like,

client := http.Client{
        Transport: &http.Transport{Dial: dialTimeout},
}

Add username to parameters map, to be configurable, refer to hdfs.go. And it can be set by env HDFS_USER_NAME in unit test.

Improve the method of parsing int parameter. I re-confirm the legal type of HDFS block size, buffer size and replication. In HDFS server is depend on JAVA, Under which they are LONG INT and SHORT. So, in golang they should be int64, int32 and int16.

@stevvooe @RichardScothern @aaronlehmann  @BrianBland
The above review comments are fixed. Please take a look. Thanks a lot!

Dear @RichardScothern @aaronlehmann , @BrianBland
I find a PR to add function CloseStreamWriter to each storage driver. I'd submit another PR to match it.
Currently I hope this PR can be accepted. Please put up your proposal freely, thanks a lot!

@xiekeyang: I have also written a webhdfs driver for the docker registry and just noticed this pull request. Since you beat me to the punch I would love to be able to use your driver and see it make its way back into the docker registry source so that it can be used by others and supported as the registry moves forward. One problem I am seeing here is that you do not handle name node failover. Our HDFS cluster has two name nodes, an active and standby node. In order to accommodate this type of HDFS system which I think is a pretty common setup there needs to be two urls passed into the driver, the active node url and the passive node url. Once you have these two nodes you will need to make sure that you are indeed making requests to the currently active node in case of failover. I would love to help out getting this logic in as I already have it implemented in my driver. That being said there are still some edge cases I need to suss out. Overall this driver looks great, hopefully we can get this feature in and I would be more than happy to help out in any way possible to get this PR in.

@jakecharland : appreciate so much! Could you please share your implementation benevolently? I'd improve my PR.

@xiekeyang: Yes the two name nodes are usually configured via the hdfs-site.xml. I did contemplate having this xml file as a dependency for the hdfs registry and then parsing it in go but I decided it was overkill since the registry is using web hdfs there is no guarantee that you are running on a machine that is configured as an hdfs client and setting it up as an hdfs client seems weird for something that is communicating over a restful api.
Anyway what would be the best way to get my code to you?

@jakecharland
you can share code on github, or send it by email. Thanks.
<EMAIL_ADDRESS>

@RichardScothern
I improve the PR, to define a http.Client in FileSystem.
it is reused in each webhdfs http connection.
In test suite, http connection number is reduced.
Please take a look, and put up your suggestion.
Thanks so much!

@RichardScothern
Do you think fault still exist in the implementation? please don't hesitate to tell me, appreciate.

@stevvooe @RichardScothern
Here HTTP client is reused. In netstat monitor the max client number is limited to 32, which may be the concurrency number of unit test. So, can we regard it as OK?
Hopefully, this feature can be contributed to distribution.

@xiekeyang I think there is a large problem in the logic for WiteStream. When WriteStream is called with an offset that is not equal to the length of the file the driver implementation will load the whole file into memory and then truncate it client side. I believe this is a pretty large flaw as when I was running the tests my comp was getting up to ~50gb of memory being used obviously lots of that is swapping. @RichardScothern @stevvooe would probably know if this is ok or not but my feeling is that it should be changed. I have implemented it in my hdfs driver without reading the whole file into memory and would be more than glad to share this with you.

Thanks.
Is your implementation located on https://github.com/jakecharland/distribution/blob/hdfs/registry/storage/driver/hdfs/hdfs.go ?
I'm reading it. After improving the function, I would inform you.
By the way, I've read your implementation by RPC protocol.
It is on https://github.com/jakecharland/distribution/blob/hdfsNative/registry/storage/driver/hdfs/hdfs.go .
You are using package github.com/colinmarc/hdfs. But as your issue to colinmarc, it seems still have some problem. And I found colinmarc/hdfs seems to be not as optimizational as distribution requirement. I've a idea that we can export the RPC protocol part of code, optimize and package it for distribution by ourselves. I think it is wonderful for distribution.

@jakecharland
In WriteStream I make usage of hdfs append request for the case of offset greater than file length.
However, for the case of offset lower than file length, I still read full hdfs file to buffer.
It is because that hdfs truncate rest api have serious problem. It must repeat some times the request to success. What's more, the repeat-interval is up to 1 sec, which is hard to be accepted by users. It is low performance and big risk.
Of course, it is my opinion, we can discuss about this freely.

@xiekeyang Ya I had the same problem with truncate as well. I still think that holding the whole file in memory before writing to disk justifies the use of it. I would at least prefer to see the file written in chunks to HDFS and not stored in one buffer and then written out at once. This optimization would at least fix a lot of the memory issues I am seeing.

@jakecharland
I understand you. Some drivers like s3 writing stream chunk by chunk is because s3 protocol limits this. If the writing length is over chunk size, the request will fail. But HDFS allow the writing length to over chunk size. In server the received data will be assigned to different chunks. Here we may refer to driver inmemory, which buffer whole reader data, and write once time.
To your worry for reading big exist file, I'm think to use hdfs open api directly to offset position, instead reading full file. It can partly resolve the problem.

@jakecharland @RichardScothern
I add the function of append to hdfs drive and update it to PR.
In WriteStream, some cases is improved:

if offset is bigger than file length, append the reader data to file directly, avoiding read the file unnecessarily.
If file exists, read the file data only to offset position and buffer it, instead reading full file data.

What is left in order to get this merged?

@timcharper a few things:

The storage driver interface has been changed and this PR needs to be updated
The issues raised in this thread to be resolved (if they aren't already)
Positive code reviews preferably from the people involved in this thread

I just rebuild the HDFS storage API functions, to fit the new storage interface.
Thanks to simpler new interface, as remove the offset argument, The defect issue above is resolved. Let me introduce:

In Writer function, driver needn't read the current file to buffer, or truncate the file any more. It just need re-create or tailed-append the file. So, a) big local buffer is avoided. b) HDFS truncate API, which is failed accepted by server, is unused. c) The performance, let's say, should be nice.

The Writer interface use HDFS block writing. The action is sliced to blocks and sent via HTTP requests. It because that in HDFS web transaction, server slice the data, and store to datanodes. So, this patch make sure nice performance, and fit to storage interface, as actions of write, commit, close.

please @RichardScothern @aaronlehmann @stevvooe  take a review, thanks so much.

@maintainers: any proposal?

@xiekeyang I build docker registry image with the branch https://github.com/xiekeyang/distribution/tree/hdfsweb ,and config hdfsweb storage like https://github.com/xiekeyang/distribution/blob/drv-hdfs/docs/storage-drivers/hdfsweb.md.But I cannot push image.please Look at the following error：
time="2016-07-01T03:33:38.840768662Z" level=error msg="response completed with error" environment=development err.code=unknown err.detail="hdfsweb: Put http://hadoop3.jd.com:50075/webhdfs/v1/myregistry/docker/registry/v2/repositories/nginx/_uploads/54b72328-0b02-4116-bd3f-89e55f046642/startedat?op=CREATE&user.name=root&namenoderpcaddress=hadoop3.jd.com:8020&blocksize=67108864&buffersize=4096&overwrite=true&permission=666&replication=1: dial tcp <IP_ADDRESS>:50075: i/o timeout" err.message="unknown error" go.version=go1.6.2 http.request.host="hadoop1.jd.com:5000" http.request.id=e48804cc-c221-4247-b6c2-438787845a20 http.request.method=POST http.request.remoteaddr="<IP_ADDRESS>:35604" http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/4.4.0-21-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.040653676s http.response.status=500 http.response.written=556 instance.id=1ac2e635-69ed-475e-a876-34a6592f9cbe service=registry vars.name=nginx version=

@xiekeyang I build docker registry image with the branch https://github.com/xiekeyang/distribution/tree/hdfsweb ,and config hdfsweb storage like https://github.com/xiekeyang/distribution/blob/drv-hdfs/docs/storage-drivers/hdfsweb.md.But I cannot push image.please Look at the following error：
time="2016-07-01T03:33:38.840768662Z" level=error msg="response completed with error" environment=development err.code=unknown err.detail="hdfsweb: Put http://hadoop:50075/webhdfs/v1/myregistry/docker/registry/v2/repositories/nginx/_uploads/54b72328-0b02-4116-bd3f-89e55f046642/startedat?op=CREATE&user.name=root&namenoderpcaddress=hadoop:8020&blocksize=67108864&buffersize=4096&overwrite=true&permission=666&replication=1: dial tcp <IP_ADDRESS>:50075: i/o timeout" err.message="unknown error" go.version=go1.6.2 http.request.host="localhost:5000" http.request.id=e48804cc-c221-4247-b6c2-438787845a20 http.request.method=POST http.request.remoteaddr="localhost:35604" http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/4.4.0-21-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.040653676s http.response.status=500 http.response.written=556 instance.id=1ac2e635-69ed-475e-a876-34a6592f9cbe service=registry vars.name=nginx version=

@5feixiang
It is likely tcp disconnection between your host pc and remote HDFS server. please check connection.
if it is OK, you can use some hdfs unit test function, or call fs.go interface directly, to confirm the good configuration. then set config.yml file according to the good configuration.
I share the hdfs interface, with example, on https://github.com/xiekeyang/webhdfs . you may refer to it.

@xiekeyang I can connect remote HDFS server at my host pc. But cannot connect remote HDFS server  in my registry container.for example telnet remote HDFS server port 50075 is warting....
Have you built registry images and executed pushing images successfully?

@xiekeyang I set hdfs storage in my config.yml file as follows:
hdfsweb:
namenodehost: <IP_ADDRESS>
namenodeport: 50070
rootdirectory: /myregistry

@xiekeyang  Why dial tcp <IP_ADDRESS>:50075: i/o timeout  ？where is ip address <IP_ADDRESS> from？

@xiekeyang  I can user your hdfs interface operating my remote hdfs,but in registry container cannot,error as follow:
Put http://hadoop:50075/webhdfs/v1/mytest/foo?op=CREATE&user.name=kxie&namenoderpcaddress=hadoop:8020&blocksize=67108864&buffersize=4096&overwrite=true&permission=666&replication=1: dial tcp <IP_ADDRESS>:50075: i/o timeout

Please add the hdfs server host name to your container's /etc/hosts, or run the registry as net host mode.

@xiekeyang I have already add the hdfs server host name to my container's /etc/hosts, the problem still exists.

@xiekeyang Thank you very much,I check that hostname of my host pc has problem.

@xiekeyang I have to apologize. I am not full time on distribution, so don't have nearly as much time to look at code. We are also very busy with the upcoming releases.
I know @jakecharland took a solid look at it back in February, but I'd like to see some deep review from @RichardScothern, as well.

@xiekeyang  If parameter replication more than one,cannot push image successfully.please Look at the following error：
Error: Status 503 trying to push repository hello-world: "{"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}\n"

@xiekeyang Why always occur errors:
time="2016-07-04T09:23:26.160587653Z" level=error msg="response completed with error" environment=development err.code="manifest unknown" err.detail="hdfsweb: Get http://HOSTNAME:50075/webhdfs/v1/tmp/registry/docker/registry/v2/r

CC @jakecharland @RichardScothern

@xiekeyang
trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Writer" trace.id=9baf83d8-8e92-4914-9152-824f542c4299 trace.line=124 vars.name=ubuntu version=
fatal error: runtime: out of memory
runtime stack:
runtime.throw(0x100c6a0, 0x16)
github.com/docker/distribution/registry/storage/driver/hdfsweb.(*driver).newWriter(0xc8203b3400, 0xc82018ec60, 0x5a, 0x0, 0x0, 0x0)
/go/src/github.com/docker/distribution/registry/storage/driver/hdfsweb/hdfs.go:425 +0x311 fp=0xc8204a0518 sp=0xc8204a0468

@xiekeyang thanks for updating with the new storage driver.  I will re-review this as soon as i get a chance.

Hi guys, sorry for necroposting, but is this going to be worked on further? What's the status here?

@ror6ax unless @xiekeyang or someone else is wiling to put the energy and time in this, this is unlikely to receive love from maintainers right now.

Implement a driver use golang native HDFS library instead of webhdfs #3123


Re-resolve DNS for Wireguard

https://wiki.archlinux.org/title/WireGuard#Endpoint_with_changing_IP

Already added in https://github.com/arch-anes/self-hosted-services/commit/e8895be68ec33fe2a8c363fd566fd6d57576059f


AttributeError: 'Browser' object has no attribute 'get_screenshot_path'

Describe the bug
take screenshot fails
*** Settings ***
Library  Browser

*** Test Cases ***
Going to my location on google maps
   New Browser   headless=${False}
   New Context   geolocation={'latitude':40.712772,'longitude':-74.006058}  permissions=['geolocation']
   New Page      https://maps.google.com
   Click     \#widget-mylocation
   wait until network is idle  timeout=10s
   take screenshot

After parsing some git history with  git log -Sget_screenshot_path -p I'm pretty sure that method never existed... I'm fixing this now and adding a minimal test.
Fixed in d9783d4c1e96636c5fb0bb23d5307013936cee86


Warning: When used inside StackLayout with vertical orientation, RadListView should have an explicit height

Environment
Provide version numbers for the following components (information can be retrieved by running tns info in your project folder or by inspecting the package.json of the project):

CLI:  "8.0.0",
Cross-platform modules:
Android Runtime:"6.0.0
iOS Runtime:"6.0.1"
Plugin(s): "7.0.1","nativescript-ui-listview"
NativeScript-Angular:6.0.1
Angular:8

Describe the bug

To Reproduce

Expected behavior
Sample project

Additional context

@NikhilAkorde
This is expected. The warning is self-explanatory - you should restrict the height of RadListView, in order to avoid bad performance, caused by RadListView having infinite size and trying to layout all of its items.


Code tab autocomplete in editor

Describe the feature
To properly complete with Copilot, this extension needs to support tab-autocomplete in the editor. I've seen other non-copilot extensions support this capability, so it's definitely possible.

This feature is still planned for this extension, but it does require a lot of work:

Need to decide what code gets included when you query the LLM.
Need to balance context vs cost.
If the user is using an expensive model like o1, we definitely do not want to use that model for the code complete. So, we need to add a setting / GUI option for what model to use for code complete.
Understand what VS Code APIs to use to show an auto-complete inside the editor.


Guideline/recommendation for ASP.NET injection of different Container instances

Hi!, I'm starting out with the SDK and would like any recommendations for my scenario.
I have an ASP.NET Core application and was thinking on injecting the appropriate Container instance into my repositories.
The problem comes from ASP.NET injection not having a way to distinguish the different Container registrations I want to make (no named registrations), for example:
services.AddSingleton<Container>(userContainer);
services.AddSingleton<Container>(ticketContainer);

// :(
// class UserRepository { UserRepository(Container container) { } }

And as I don't control the creation of the Container directly but it's done through the CosmosClient, I cannot subclass it as a way to resolve the ambiguity on my repositories.
To get around this I've created a wrapper for Container and sub-classed it for each specific one:
// Something like this:
// class ContainerWrapper : Container { ContainerWrapper(Container container) { } }
// class UserContainer : ContainerWrapper
// class UserRepository { UserRepository(UserContainer container) { } }

services.AddSingleton<UserContainer>(new UserContainer(userContainer));
services.AddSingleton<TicketContainer>(new TicketContainer(ticketContainer));

This works well enough but it's cumbersome as my wrapper basically has to do a pass-through of all Container methods which is obviously not ideal.
I suppose it's more of a limitation of the DI container used, but would like to know your thoughts on how to deal with this.

I have the preference of  injecting the Container object because it seems cleaner to me, as opposed to injecting the CosmosClient or a factory function for Container. Do you think this is a good idea?
Do you see a better way to achieve this?
Is it a good idea to keep the Container objects as singletons or what are the recommended lifetime for them and CosmosClient?
Any other recommendations based on this scenario?

@liquidsnk My take on DI and Singletons is that you would mainly use it to store objects that are reusable and expensive to create, or where (like the CosmosClient) it is recommended to maintain a single instance to benefit from how the object behaves (caching request routing like the CosmosClient).
In the case of the Container, the Container has no cost, so using DI doesn't make much sense. It would be arguable that the consumer of the DI instance doesn't need to know the container name (a string), but they are indirectly knowing it (they are asking for a specific container).
I would just keep the CosmosClient in DI, and call GetContainer when needed on the Controllers. They already know which container they want to access. You could have a helper class if you don't want to have strings around, that would be like:
public static ContainerHelper
{
    private static string database = "YourDatabase"; // This would also come from IConfiguration and not have a static class?
    private static string userContainer = "users";
    private static string ticketsContainer = "tickets";

    public static Container GetUserContainer(CosmosClient client) => client.GetContainer(ContainerHelper.database, ContainerHelper.userContainer);
    public static Container GetTicketsContainer(CosmosClient client) => client.GetContainer(ContainerHelper.database, ContainerHelper.ticketsContainer);
}

It can be improved by leveraging IConfiguration, but serves as a rough example.

Thanks you for your answer!. Is good to know that Container is currently a lightweight construct.
I like the option of being able to provide instances with what they need to work and no more, they don't even need to be aware of the lifetime of Container as long as it's equal to or exceeds their own.
This might be very dependent on how these repositories are designed however (one per container), and I'm not entirely against injecting CosmosClient instead, so I'll keep it in mind in case the wrapper thing becomes unbearable!

@liquidsnk I share your thoughts on this.
You can support this entirely in the DI layer if you are happy to use implementation factory methods. Something like:
services.AddSingleton(_ => new CosmosClient(settings.CosmosDbConnectionString));

services.AddTransient<IUserService>(sp =>
{
    var cosmosClient = sp.GetRequiredService<CosmosClient>();
    var container = cosmosClient.GetContainer(settings.databaseName, "Users");

    return new UserService(container);
});

services.AddTransient<ITicketService>(sp =>
{
    var cosmosClient = sp.GetRequiredService<CosmosClient>();
    var container = cosmosClient.GetContainer(settings.databaseName, "Tickets");

    return new TicketService(container);
});

I like this approach, as it avoids having to introduce additional abstractions purely for the purposes of DI. I think CosmosClient.GetContainer is a lightweight proxy, so is OK to construct the containers from the CosmosClient on each lifetime in this way.
On the flip side, it does mean that you need to get all dependencies for each service and pass to the service ctors yourself. Only real issue with this is having to update the DI factory methods when the service dependencies change (e.g. another dependency is added).


Populate Gateway Status address for ClusterIP type Envoy Service

Description:
Relates to https://github.com/envoyproxy/gateway/pull/1205#issuecomment-1483594260

pls assgin to me


fix: week number is actually Monday based by default

🐛 Bug Fix
Most databases, including ISO standards, actually uses Monday as the first day of a week. The default week time grain in Superset (P1W) should translate to Monday-based weeks, too.

Thanks I checked presto and it looks like the logic for presto is date_trunc('week', CAST({col} AS TIMESTAMP)) and when I tested it out it looks like week starts on Monday. Do we know if this is true for all supported db_engine_specs? It seems like if any of them evaluate to week start Sunday, this would be incorrect. Can we just show the date for the start of the week (to make sure we will always be accurate)?

Can we just show the date for the start of the week (to make sure we will always be accurate)?

That was fixed in #486 .

Do we know if this is true for all supported db_engine_specs?

I also checked Druid, MySQL, PostgreSQL, SQL Lite and some other databases, with the exception of MySQL and Drill, all others are using Monday-based weekdays.
Given how most businesses operate and that Monday-based is also the ISO standard, I propose we move forward with this change and change MySQL & Drill to be Monday-based, too.
cc @mistercrunch @john-bodley @kristw

Another option is to simply remove the week numbers, as: 1) it is not useful for most business settings anyway; 2) different businesses may have different definition of week-of-years.

I agree with removing Wxxx

Thanks for resolving this! This seems like a good solution.


Tokens filter mismatch error when routing

I ran into some issues with routing, and I am not sure I am doing something wrong, or this is an expected behaviour...
I am trying to define a catch-all route in py4web that will handle things not handled by any other route. Here a minimal code example:
# __init__.py
from py4web import action

@action('index')
@action('<project_id:int>')
def index(project_id=None):
    return f"Index: (project_id={project_id})"

@action('<anything:path>')
def catch_all(anything):
    return f"Catch-all: {anything}"

If I try to access the application with some random URL, I get a pretty extensive traceback with the main error message being:
ombott.router.radidict.RadiDictKeyError: tokens filter mismatch

The problem is reported while processing the:
@action('<anything:path>')

Not sure, if what I try is not valid or otherwise wrong, or it is a bug in Ombott?

Here is an answer I got from Val K elsewhere:

Hi!
Short story: try to add static prefix to the first route: @action('projects/<project_id:int>')

Long story: ombott uses radix-tree hybrid router, it is declaration-order-independent and instead of order-dependent resolving it prioritizes static route-fragment over dynamic one, since this is most expected behavior.
So this solution has some constraints  - you can not have 2 (or more) routes that have dynamic fragment of different types (int, path) in the same place.

Hence the behaviour is expected and it is a documentation issue rather than a bug. I submitted a PR (https://github.com/web2py/py4web/pull/938) to improve the documentation, and I am closing this issue.


Can we not convert null to empty string?

Can we not do this? https://github.com/erikras/redux-form/blob/b531cbeb72f97b85771845664ac68bf2da8360c7/src/createFieldProps.js#L53
When I pass null, I expect null, not an empty string.

I think it i intentionally behaviour.
As mentioned in https://github.com/facebook/react/releases/tag/v15.0.0:

React 15 warns you on a null input value and offers you to clarify your intention. To fix the warning, you may explicitly pass an empty string to clear a controlled input, or pass undefined to make the input uncontrolled.

If you need null value for your own Field component, I suggest to use parse prop:
<Field
  name="priority"
  component={NumberInput}
  format={(value) => (value === '' ? null : Number(value))}
  min={0}
  max={10}
/>

Sure, thanks.

Published in v6.2.1.


trans body with get method

Unirest.get() can not trans body.is it correct？

No,
Because the HTTP spec is very specific that servers should not honor any body on a GET
https://tools.ietf.org/html/rfc7231#page-24

A payload within a GET request message has no defined semantics; sending a payload body on a GET request might cause some existing implementations to reject the request.

I'm going to go ahead and close this, feel free to follow up in the new github discussions if you want to chat more. https://github.com/Kong/unirest-java/discussions


backup_configuration.location is not supported

I see the following with terraform plan
          ~ backup_configuration {
              - location                       = "us" -> null
            }

Currently, there is no way to set this param with the module.

Closing due to repo sunset


git - implement distributed part?

currently abapGit does not implement the distributed part of git, it has to have a connection to the git server, and pulls all information from the server each time.
However, normal git clients store all old objects/files(blobs) locally.
Perhaps it is time to look into if this is a good idea for abapGit? It will give better performance for large repositories, as all objects does not have to be fetched each time.

Not crazy but cool imho :)
Some practical highlights around this (not directly related but about performance)

One of the annoying thing now is actually local serialization. E.g. I want to see diff - it serialize complete repo, then I press "back" -  it serialize complete repo again. For the big project it takes time. I actually have some ideas how to improve it ... some day ... (may come sooner if I'll have to deal with that big project more in near future ;)
There is a practical issue with abap which does not meet much in other git-relevant environments. Many developenrs - one environment. 2 people cannot work on 2 feature at the same time. For me this is an issue. Some kind of stashing may help - e.g. saving changed content when swicthing branches.

the functionality is implemented where it caches the serialized objects. However the code calls refresh quite often to make sure everything is the latest version.

There is only 1 active version of each program/class in an ABAP system, we cannot change this. Stashing: why not just commit the code to a new branch before switching?

I'll make some coding to show what I mean :)

Yes, I meant that this is an inconvenience point worth to thinking on. Stashing: genearlly agree. But what if it was not commited and then person 2 wants to continue his work and don't know what to do with the existing changes. Automatic stashing could be a nice feature to protect from mistakes. Anyway just as an idea.

closing, not in scope for now


App crashes since connected to NS server

Your Environment

Uploader Version Number: 0.7.3
Your Android Phone Model Name (e.g. Samsung J1 Ace): Samsung Galaxy S4
Android Version Number (e.g. use 4.4 for 4.4.4): 5.0.1
Network Connection at the time the issue occurred (Wi-Fi, Mobile): Wifi, not sure about mobile
(if relevant) Nightscout website version (e.g. 0.8.4 Funnel cake, master / dev branch): 14.2.2

Brief Explanation of Issue

Occasionally the app stops. (German Error message from OS is "600 Series Uploader angehalten."
I've been using the app for month offline (without NS, just local xDrp+) without any problems. Now i configured my own NS installation (on Linux using Docker) and since then the app crashes sometimes (between every 5 minutes and 8 hours)
Uploader Error Code

None visible, but if you tell me how, I can send you the logs (already set to debug).
Steps to Reproduce (for bugs)

Severity Score

4 (no data until app restarted)

Can you try version 0.7.1  and see how you get on.

I will give it a try when the boy is home again.
Could the HTTPS connection to NS be the reason? I think it all started after I activated HTTPS on the NS server... I will try that first and report back to you.

HTTP didn't change anything. I thought this could help because I had a test server running with HTTP on my PC. It worked without problem there (maybe just not long enough, maybe a coincidence).
When migrating to my server with HTTPS the problem started after some time.
I downgraded to 0.7.1 now. Even though I miss the "tap title to switch graph/logs" feature ;)
I'll report back when I know if 0.7.1 works properly.

You can check the https settings on your server through this site https://www.ssllabs.com/ssltest/

SSL is ok. I have the certificates running on my webserver for quite some time. The problem persists when disabling HTTPS, so - as mentioned above - it is not an SSL issue.
For my webserver, the ssl-test gives grade B. No errors, just warnings about no PFS and TLS 1.0/1.1 support.
The only errors where when simulating handshake with IE on Windows XP - I can live with that.
I was not able to check the NS directly as the ssl-check does not allow custom ports. The test was performed against my apache using the same certificates.

I'm sure you have the correct certificates, I said that the setting on the web server (Apache in your case) might not be correct. Check the logs on your web server. Also, you can see how I setup the NS on my server (sorry for Russian).


XSD data types instead of xs:string for value type

I was wondering why, for value, the type is always xs:string and we use 'valueType' with DataTypeDefXsd/* as value?
If everything is a string, then how do I query over property values? Either I should manually convert the types or infer the types automatically. But in the provided ontology, DataTypeDefXsd does not contain any linking to the actual XSD type.

Example Double property serializaiotion
@prefix aas: <https://admin-shell.io/aas/3/0/> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xs: <http://www.w3.org/2001/XMLSchema#> .

<something_48c66017> rdf:type aas:Submodel ;
    <https://admin-shell.io/aas/3/0/Identifiable/id> "something_48c66017"^^xs:string ;
    <https://admin-shell.io/aas/3/0/Submodel/submodelElements> [
        rdf:type aas:Property ;
        <https://admin-shell.io/aas/3/0/Referable/idShort> "something3fdd3eb4"^^xs:string ;
        <https://admin-shell.io/aas/3/0/Property/valueType> <https://admin-shell.io/aas/3/0/DataTypeDefXsd/Double> ;
        <https://admin-shell.io/aas/3/0/Property/value> "1234.01234"^^xs:string ;
    ] ;
.

One alternative approach:
instead of <https://admin-shell.io/aas/3/0/Property/value> "1234.01234"^^xs:string ; we use <https://admin-shell.io/aas/3/0/Property/value> "1234.01234"^^xs:double; and remove <https://admin-shell.io/aas/3/0/Property/valueType> <https://admin-shell.io/aas/3/0/DataTypeDefXsd/Double> ; because it is redundant.
@prefix aas: <https://admin-shell.io/aas/3/0/> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xs: <http://www.w3.org/2001/XMLSchema#> .

<something_48c66017> rdf:type aas:Submodel ;
    <https://admin-shell.io/aas/3/0/Identifiable/id> "something_48c66017"^^xs:string ;
    <https://admin-shell.io/aas/3/0/Submodel/submodelElements> [
        rdf:type aas:Property ;
        <https://admin-shell.io/aas/3/0/Referable/idShort> "something3fdd3eb4"^^xs:string ;
        <https://admin-shell.io/aas/3/0/Property/value> "1234.01234"^^xs:double;
    ] ;
.

As discussed in IDTA Open Hours, the reason was because of more flexibility, avoiding information loss in round trip conversions, and stay consistent with XML/JSON mapping.

@sebbader Regarding the datatypes, as it was mentioned, the string type chosen for all types. However, in DataSpecificationIec61360 the level type has boolean datatype:
https://github.com/admin-shell-io/aas-specs/blob/bb6747a17ca51d3a08bcf03b2fc927671ab3433b/schemas/rdf/examples/generated/DataSpecificationIec61360/maximal.ttl#L57-L63
Is this an inconsistency or there is a reason for it?

Furthermore in Blob we are also using xsd:base64Binary
https://github.com/admin-shell-io/aas-specs/blob/13b904f5b546aade0459b40e3b30953e6af50e0a/schemas/rdf/examples/generated/Blob/maximal.ttl#L76

@kenwenzel Do you have any idea regarding the usage of datatypes and also the inconsistencies?

@mhrimaz just a remark related to these changes: please make sure that you patch aas-core-codegen, not the schemas directly.

I also agree that this is not idiomatic RDF and has many problems.

Semantic repositories index number and date literals specially, so that you can make fast comparisons and range searches.
If you keep the datatype in a separate field, that functionality is lost
For modern XSD dateTime stamps (year 0000..9999) without timezone, that's not a big loss since lexicographic string sort does the same
But for numbers that is a big loss
You cannot even check equality of values, unless they are normalized. "3.14"^^xsd:decimal = "03.140"^^xsd:decimal but that is not true of strings

more flexibility

More flexibility at the loss of defined functionality is no win at all.

Why do we use strings? It would be more flexible to break them down into constituent characters and store them in individual nodes: then I can point to an individual character, right?
The reason is that everyone would hate such design decision: it's impractical

Why no XML attributes?

I find the arguments here weak.

XML defined XSD Datatypes and xml:lang maybe 20 years ago. There's no good argument why these standards should not be followed
Ok: do not use custom XML attributes if you suspect those will need to be expanded
But do use attributes that have been standardized 20 years ago!

avoiding information loss in round trip conversions, stay consistent with XML/JSON mapping

There is no information loss!

In XML just use the predefined attributes
In JSONLD the fields are predefined (@value, @type) but can be renamed in the context

in DataSpecificationIec61360 the level type has boolean datatype
in Blob we are also using xsd:base64Binary

The reason is that these props are predefined (fixed) in the spec.
But such rational use is not permitted for props to be defined in a particular AAS application ;-)

@mristin @mhrimaz @sebbader  @kenwenzel @wiresio
https://github.com/aas-core-works/aas-core-codegen/blob/94399e10457ccccf5fb4ff4c7bfb714a80c7340d/test_data/rdf_shacl/test_main/expected/aas_core_meta.v3/expected_output/rdf-ontology.ttl#L541
aas:DataTypeDefXsd defines its own URLs so it disregards the XSD Datatypes standard, which was defined:

https://www.w3.org/TR/2000/WD-xmlschema-2-20000225/ first 24y ago
https://www.w3.org/TR/xmlschema-2/ in its final form 20 years ago

I think it is a bad practice to disregard age-tested standards, especially those that are the basis of typed values in both XML and RDF.
Eg If we take this as example:
https://github.com/admin-shell-io/aas-specs/blob/2ab08f92bdd1d44edc1cfee52552fe5429d2178e/schemas/rdf/examples/generated/Property/OverValueExamples/Any_URI/relative_path_without_scheme.ttl#L9-L14
it should be changed to the following (skipping ^^xs:string, which is superfluous)
    <https://admin-shell.io/aas/3/0/Submodel/submodelElements> [
        rdf:type aas:Property ;
        <https://admin-shell.io/aas/3/0/Referable/idShort> "something3fdd3eb4";
        <https://admin-shell.io/aas/3/0/Property/value> "path/to/somewhere"^^xs:anyURI;
    ] ;

Can you point me to some JSON examples? Then we can figure out how to enrich the context to convert the JSON to this desired RDF representation.

@sebbader Regarding the datatypes, as it was mentioned, the string type chosen for all types. However, in DataSpecificationIec61360 the level type has boolean datatype:
https://github.com/admin-shell-io/aas-specs/blob/bb6747a17ca51d3a08bcf03b2fc927671ab3433b/schemas/rdf/examples/generated/DataSpecificationIec61360/maximal.ttl#L57-L63
Is this an inconsistency or there is a reason for it?

I assume this is a bug: in former pre-releases this was an enumeration.


chore: remove CodeQL

Reviewers
Functional:
@atolopko-czi @MDunitz @maniarathi
Readability:

Changes

Removes the CodeQL workflow.
Might need some additional changes in GitHub to deactivate the related checks.

Need to update the Github Action as well to not run it, I think.

Need to update the Github Action as well to not run it, I think.

GHA will use this file to run it, so deleting it should be enough. It is not referenced anywhere else


src/math.rs is empty?!

It's weird that this file contains zero lines.
Also, I tried building the doc, that failed and several errors showed up stating that math* is missing.
I doubt this is a coincidence. Maybe an error uploading the file?

This file should not exist, I will remove it.
Building the documentation with cargo doc will invariably fail with unresolved import errors. You have to select the variant of nphysics you want to build the doc for. For example for the 3d simulator using 64 bit floating point values, you need to use cargo doc --features "3df32".

Oh okay, this makes sense. Thanks!


Some Discord webhooks do not work for me

What version of Node.js are you using?
v16.18.1
What version of npm are you using?
8.19.2
What operating system are you using?
Linux Debian 11
What version of SnailyCAD are you currently using?
Newest commit (80e6c57)
Describe the Bug
Unfortunately, the following webhooks do not trigger Discord notification for me:

Panic

citizen records webhook I have not tested yet.
Expected Behavior
That a Discord notification is sent that a panic button has been triggered.
To Reproduce
Set Webook Channel for Panic under: Admin -> CAD Settings -> Discord Webhooks -> "Panic button channel".

Are you able to share the channel integrations tab? Where all the webhooks are listed, I want to see if there's any duplicated ones or unnecessary ones.

Sure,

Sincerely

Are you able to test other webhooks? Just to make sure it's only the panic button webhook

Okay, I have tested it.
All webhooks work except Panic.
Sincerely

Fixed now!

Good evening, I have now tested it again but unfortunately again without success.
My friend has also tried it.

Then I have to click again on Officer -> Dashboard and can then trigger the Panic Button.
But the Discord notification still doesn't work for my friend and me.
I have already tried to delete the webhook in SnailyCAD, then save it and set it again. But even then no Discord message comes.
Translated with www.DeepL.com/Translator (free version)

Does it work if you select the panic button ten-code? Not the "Panic Button" button?

Thank you very much, it works now.
I had to create another 10-code panic button which is connected to the original "Panic Button" button.
Now as soon as you toggle one of the two it works.
Sincerely.


Alternative Linux (iptables) method to resolve in host

Abusing a bit the loopback interface, we can achieve resolution in the host without messing with dnsmasq nor resolv.conf
Tested on Debian 8 (jessie).
Steps:

Add a local DNS server (we'll choose <IP_ADDRESS> to avoid collitions) to your network connection, either:

If using DHCP networking, edit /etc/dhcp/dhclient.conf and add prepend domain-name-servers <IP_ADDRESS>;
If using static networking, add <IP_ADDRESS> to your network connection via Network Manager (or other means)

Redirect DNS queries on that loopback interface to landrush daemon. Use the following iptables rules:

iptables -t nat -A OUTPUT -p udp -d <IP_ADDRESS> --dport 53 -j DNAT --to <IP_ADDRESS>:10053
iptables -t nat -A OUTPUT -p tcp -d <IP_ADDRESS> --dport 53 -j DNAT --to <IP_ADDRESS>:10053

Pros:

This may be automated inside landrush, providing a more straightforward integration in Linux

Cons:

This method intercepts DNS queries globally, not only for selected domains

+1


Added sass @use functionality.

Updated the import regex to also handle sass's @use directive, as described in #47.

This may create a secondary problem. CodePen uses PostCSS Use (it predates Sass’s use) to use plugins. AFAIK, the two systems are wholly incompatible, as they reached for the same namespace for completely different reasons. In my opinion, Sass has the advantage here, as they are a superset of CSS, whereas this one plugin is not making the same claim.
The problem, for me, is that I could introduce a feature that breaks a website like CodePen, because folks reach for @use and CodePen errors out because it’s not a valid plugin.

Oh no, I had no idea that a separate @use directive existed for postcss plugins. That is a problem. In addition, I've noticed that while the fix does allow for the plugin to be compatible with the new syntax, it ends up causing errors anyway because the CSS is actually injected before the SASS parser begins to process the stylesheets. This will require a different solution I believe. Thanks for the quick feedback though. :)

Closing this with the resolution that it will require a different solution.


More examples is required

Really, one example is too little for the library. I'll be grateful if you add some extra examples..

@vie10 i concur, i am planning to work on documentation on each feature with examples, by this weekend.
Thanks for raising the issue.

@vie10 i concur, i am planning to work on documentation on each feature with examples, by this weekend.
Thanks for raising the issue.

I need just a messaging pipeline between applications. How I can do it with the library? Looks like I need just a stream

@vie10 i concur, i am planning to work on documentation on each feature with examples, by this weekend.
Thanks for raising the issue.

I need just a messaging pipeline between applications. How I can do it with the library? Looks like I need just a stream I'll be grateful if you can help me..

I already figured it out so keep your time.


HARD challenges mass rename

Update the hard challenge names to be compliant with the new naming scheme.
Interestingly, there was a "166b" challenge. I didn't know if this was a typo, so I named it "... #0166b ..."

Looks good! Refs #20. Merging...


How to redirect it to homepage instead of showing this in new tab.

{"result":"success","row":2}
How to redirect it to homepage instead of showing this in new tab.

What worked for me was:

set the Web App access/auth, "Who has access" to Anyone.
you can do this in appscript by clicking Deploy, Manage Deployments, edit

then in your html file after closing  tag.