Collection of a number of important efficiency improvements

This PR collects a number of important efficiency improvements, and a few features that were tightly bound to these efficiency changes, so they are shipped together.
In particular:

objects are now sorted and returned in the order in which they are on disk, with an important performance benefit. Fixes #92
When there are many objects to list (currently set to 9500 objects, 10x the ones we could fit in a single IN SQL statement), performing many queries is slow, so we just resort to listing all objects and doing an efficient intersection (if the hash keys are sorted, both iterators can be looped over only once - fixes #93)
Since VACUUMing the DB is very important for efficiency, when the DB does not fit fully in the disk cache, clean_storage now provides an option to VACUUM the DB. VACUUM is also called after repacking. Fixes #94
We implement now a function to perform a full repack of the repository (fixes #12). This is important and needed to reclaim space after deleting an object
For efficiency, we have moved the logic from an export function (still existing but deprecated) to a import_objects function
Still for efficiency, now functions like pack_all_loose and import_objects provide an option to perform a fsync to disk or not (see also #54 - there are still however calls that always use - or don't use - fsync and full_fsync on Mac). Also, add_objects_to_pack allows now to choose if you want to commit the changes to the SQLite DB, or not (delegating the responsibility to the caller, but this is important e.g. in import_objects: calling commit only once at the very end gives a factor of 2 speedup for very big repos).
A number of functions, including (but not exclusively) import_objects provide a callback to e.g. show a progress bar.
a CallbackStreamWrapper has been implemented, allowing to provide a callback (e.g. for a progress bar) when streaming a big file.
a new hash algorithm is now supported (sha1) in addition to the default sha256 (fixes #82). This is faster even if a bit less robust. This was also needed to test completely some feature in import_objects, where the logic is optimised if both containers use the same algorithm. By default is still better to use everywhere sha256, also because then all export files that will be generated will use this algorithm and importing will be more efficient.
tests have been added for all new functionality, achieving again 100% coverage

As a reference, with these changes, exporting the full large SDB database (6.8M nodes) takes ~ 50 minutes:
6714808it [00:24, 274813.02it/s]
All hashkeys listed in 24.444787740707397s.
Listing objects: 100%|████████| 6714808/6714808 [00:06<00:00, 978896.65it/s]
Copy objects: 100%|███████████| 6714808/6714808 [48:15<00:00, 2319.08it/s]
Final flush: 100%|████████████| 63236/63236 [00:07<00:00, 8582.35it/s]
Everything re-exported in 2960.980943918228s.

I think this is a excellent result. This can be compared to:

~10 minutes to copy the whole 90GB, or ~15 minutes to read all and validate the packs. We will never be able to be faster than just copying the pack files, and we are only 3x slower.
~2 days to just list all files in the old legacy AiiDA repo (or all objects if they are loose), and this does not take into account the time to rewrite everything, probably comparable. So we are almost 2 orders of magnitude faster than before.

Anybody brave enough to review this PR? :-)
I'm happy to explain the changes in a call, if it helps.

@chrisjsewell If possible I prefer merging (rather than squashing or rebasing) since I've been referencing the individual commits in multiple issues

Heya, gradually getting through all the code, I'm just going to update notes here as I go

mention container.clean_storage() in basic usage

delete empty folders after container.clean_storage()

as a standalone library, integrated metadata storage would be ideal (e.g. file format, file path)

drop python 3.5

use type annotations

use * before key-word arguments

use pathlib over os.path

specify "utf8" for config JSON read/write?

checking the size of the pack every time is maybe not performant: can we memoize full packs? (I guess you'd have to be careful to remove a pack from the set, if you removed anything from it):

tests/test_benchmark.py::test_pack_write
ncalls  tottime percall cumtime percall filename:lineno(function)
1       0.0000  0.0000  0.4833  0.4833  disk-objectstore/disk_objectstore/container.py:1556(add_objects_to_pack)
1       0.0484  0.0484  0.4670  0.4670  disk-objectstore/disk_objectstore/container.py:1297(add_streamed_objects_to_pack)
10001   0.0180  0.0000  0.2115  0.0000  disk-objectstore/disk_objectstore/container.py:242(_get_pack_id_to_write_to)
20003   0.1005  0.0000  0.1005  0.0000  ~:0(<built-in method posix.stat>)
10002   0.0145  0.0000  0.0799  0.0000  disk-objectstore/disk_objectstore/container.py:226(_get_pack_path_from_pack_id)

Documention notes:

Current experience (with AiiDA) shows that it's actually not so good to use two
levels of nesting.

out of interest, why?

the move operation should be hopefully a fast atomic operation on most filesystems

are there any you know of where this is not the case?

Profiling:
$ tox -e py37 -- --benchmark-only --benchmark-cprofile=cumtime

tests/test_benchmark.py::test_has_objects
ncalls  tottime percall cumtime percall filename:lineno(function)
1       0.0048  0.0048  0.4209  0.4209  disk-objectstore/disk_objectstore/container.py:778(has_objects)
10001   0.0322  0.0000  0.4123  0.0000  disk-objectstore/disk_objectstore/container.py:451(_get_objects_stream_meta_generator)
6       0.0000  0.0000  0.0896  0.0149  disk-objectstore/.tox/py37/lib/python3.7/site-packages/sqlalchemy/orm/query.py:3503(__iter__)
6       0.0000  0.0000  0.0886  0.0148  disk-objectstore/.tox/py37/lib/python3.7/site-packages/sqlalchemy/orm/query.py:3528(_execute_and_instances)
8       0.0000  0.0000  0.0874  0.0109  disk-objectstore/.tox/py37/lib/python3.7/site-packages/sqlalchemy/engine/base.py:943(execute)
7       0.0000  0.0000  0.0872  0.0125  disk-objectstore/.tox/py37/lib/python3.7/site-packages/sqlalchemy/sql/elements.py:296(_execute_on_connection)
7       0.0001  0.0000  0.0872  0.0125  disk-objectstore/.tox/py37/lib/python3.7/site-packages

tests/test_benchmark.py::test_list_all_loose
ncalls  tottime percall cumtime percall filename:lineno(function)
1       0.0000  0.0000  0.0000  0.0000  ~:0(<method 'disable' of '_lsprof.Profiler' objects>)

tests/test_benchmark.py::test_list_all_packed
ncalls  tottime percall cumtime percall filename:lineno(function)
1       0.0000  0.0000  0.0000  0.0000  ~:0(<method 'disable' of '_lsprof.Profiler' objects>)

tests/test_benchmark.py::test_loose_read
ncalls  tottime percall cumtime percall filename:lineno(function)
1       0.0019  0.0019  0.1003  0.1003  disk-objectstore/disk_objectstore/container.py:803(get_objects_content)
1001    0.0046  0.0000  0.0927  0.0001  disk-objectstore/disk_objectstore/container.py:451(_get_objects_stream_meta_generator)
1000    0.0321  0.0000  0.0321  0.0000  ~:0(<built-in method io.open>)

tests/test_benchmark.py::test_pack_read
ncalls  tottime percall cumtime percall filename:lineno(function)
1       0.0089  0.0089  0.1700  0.1700  disk-objectstore/disk_objectstore/container.py:803(get_objects_content)
10001   0.0300  0.0000  0.1407  0.0000  disk-objectstore/disk_objectstore/container.py:451(_get_objects_stream_meta_generator)
10001   0.0146  0.0000  0.0677  0.0000  disk-objectstore/disk_objectstore/utils.py:922(detect_where_sorted)
20004   0.0051  0.0000  0.0498  0.0000  ~:0(<built-in method builtins.next>)
10001   0.0030  0.0000  0.0447  0.0000  disk-objectstore/.tox/py37/lib/python3.7/site-packages/sqlalchemy/engine/result.py:1006(__iter__)
10001   0.0071  0.0000  0.0417  0.0000  disk-objectstore/.tox/py37/lib/python3.7/site-packages/sqlalchemy/engine/result.py:1320(fetchone)
20000   0.0099  0.0000  0.0253  0.0000  disk-objectstore/disk_objectstore/utils.py:385(_update_pos)
10000   0.0093  0.0000  0.0245  0.0000  disk-objectstore/disk_objectstore/utils.py:365(__init__)
10001   0.0028  0.0000  0.0237  0.0000  disk-objectstore/.tox/py37/lib/python3.7/site-packages/sqlalchemy/engine/result.py:1213(_fetchone_impl)
10001   0.0209  0.0000  0.0209  0.0000  ~:0(<method 'fetchone' of 'sqlite3.Cursor' objects>)
10000   0.0059  0.0000  0.0204  0.0000  disk-objectstore/disk_objectstore/utils.py:400(read)
20000   0.0154  0.0000  0.0154  0.0000  ~:0(<method 'tell' of '_io.BufferedReader' objects>)
10000   0.0080  0.0000  0.0109  0.0000  disk-objectstore/.tox/py37/lib/python3.7/site-packages

tests/test_benchmark.py::test_loose_write
ncalls  tottime percall cumtime percall filename:lineno(function)
1       0.0018  0.0018  0.5730  0.5730  disk-objectstore/tests/test_benchmark.py:28(write_loose)
1000    0.0039  0.0000  0.5710  0.0006  disk-objectstore/disk_objectstore/container.py:842(add_object)
1000    0.0051  0.0000  0.5671  0.0006  disk-objectstore/disk_objectstore/container.py:851(add_streamed_object)
1000    0.0150  0.0000  0.4302  0.0004  disk-objectstore/disk_objectstore/utils.py:158(__exit__)
1000    0.0079  0.0000  0.2387  0.0002  disk-objectstore/disk_objectstore/utils.py:852(safe_flush_to_disk)
2000    0.0052  0.0000  0.1639  0.0001  disk-objectstore/disk_objectstore/utils.py:870(<lambda>)
2000    0.1587  0.0001  0.1587  0.0001  ~:0(<built-in method posix.fsync>)
1000    0.0051  0.0000  0.1083  0.0001  disk-objectstore/disk_objectstore/utils.py:141(__enter__)
2000    0.1044  0.0001  0.1044  0.0001  ~:0(<built-in method io.open>)

tests/test_benchmark.py::test_pack_write
ncalls  tottime percall cumtime percall filename:lineno(function)
1       0.0000  0.0000  0.4961  0.4961  disk-objectstore/disk_objectstore/container.py:1556(add_objects_to_pack)
1       0.0522  0.0522  0.4945  0.4945  disk-objectstore/disk_objectstore/container.py:1297(add_streamed_objects_to_pack)
10001   0.0193  0.0000  0.2252  0.0000  disk-objectstore/disk_objectstore/container.py:242(_get_pack_id_to_write_to)
20003   0.1072  0.0000  0.1072  0.0000  ~:0(<built-in method posix.stat>)
10002   0.0150  0.0000  0.0848  0.0000  disk-objectstore/disk_objectstore/container.py:226(_get_pack_path_from_pack_id)


Enhancement/Request: timeout when loading XML map

The mapbox loading into a view can cause an app to hang for half a second or so. When you declare the map in XML and navigate to the page you have no way to delay that execution so it doesn't cause an interruption to the navigation transition.
A simple solution would be to add a property on the XML view, such as delayMapInit and then accept a value in milliseconds, so then we can make sure the navigation transition has completed before the map constructor is started which isn't the smoothest process. Let me know your thoughts @EddyVerbruggen, you might have a better idea since you know the SDKs better than I for mapbox.

Great idea!

Update: got Android behaving nicely locally.

looking forward to it... prepping for publish and this is one of the last things on my personal issues 😄

Can you update the plugin from GitHub (see the last check-in). You get to set a delay property in the XML definition. For instance, delay="500" means the plugin will wait half a second before invoking the Mapbox SDK.

Yes I'll test this out soon.
On Wed, Nov 2, 2016, 3:53 PM Eddy Verbruggen<EMAIL_ADDRESS>wrote:

Can you update the plugin from GitHub (see the last check-in). You get to
set a delay property in the XML definition. For instance, delay="500"
means the plugin will wait half a second before invoking the Mapbox SDK.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/EddyVerbruggen/nativescript-mapbox/issues/52#issuecomment-257995487,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFulhAH2_tNmRlIPc-0cCRcjXBRAKMb5ks5q6PgzgaJpZM4Kmnls
.

Good to go 👍

Sweet! 2.2.2 is on npm.


[stable/prometheus-operator] Add Helm Tests

What this PR does / why we need it:
TESTS!
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[ ] DCO signed
[ ] Chart Version bumped
[ ] Variables are documented in the README.md
[x] Title of the PR starts with chart name (e.g. [stable/mychartname])

/retest


Upgrade jetty version because of vulnerability

Resolves: https://github.com/stargate/stargate/security/dependabot/pom.xml/org.eclipse.jetty:jetty-io/open
Dependabot kept upgrading to a major version which breaks our code so I've manually created a PR to resolve the issue.

CI has error like this:
Step #4 - "cassandra-3.11": INFO  [Exec Stream Pumper] 2021-04-21 14:31:39,957 ProcessRunner.java:78 - stargate12-0> WARN  [dw-264] 2021-04-21 14:31:39,956 HttpChannel.java:677 - /v2/schemas/keyspaces/ks_addRow_1619015498749/tables
Step #4 - "cassandra-3.11": INFO  [Exec Stream Pumper] 2021-04-21 14:31:39,957 ProcessRunner.java:78 - stargate12-0> java.lang.NoSuchMethodError: org.eclipse.jetty.server.Request.setAsyncSupported(ZLjava/lang/String;)V
Step #4 - "cassandra-3.11": INFO  [Exec Stream Pumper] 2021-04-21 14:31:39,957 ProcessRunner.java:78 - stargate12-0> 	at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:57)

Do we need to bump DW version too?

Most likely :(

What's DW?

What's DW?

Dropwizard

Dropwizard was using an internal API: https://github.com/eclipse/jetty.project/issues/5496

@mpenick @dougwettlaufer @dimas-b I took the liberty to align the io.dropwizard.metrics dependencies in the PR and also took out the metrics-ganglia dep which is not used at all..
Seems I should update the licenses-report.txt as well, or? Any idea what's used for that?

Seems I should update the licenses-report.txt as well, or? Any idea what's used for that?

Looks like a generated file? It hasn't been updated in a bit. Let's do that in another PR.


How is the exercise effect of section 4.6 achieved

Sorry I am confused about your question

Could you clarify?

dear Marius Juston：
Sorry for the late reply. I have already mastered the usage of the UWB-Jackal-World project. Your team's projects, such as UWB-Localization, are all excellent work. Thank you for your contributions.
Thanks！
------------------ 原始邮件 ------------------
发件人:                                                                                "AUVSL/UWB-Jackal-World"                                                                                @.>;
发送时间: 2024年12月3日(星期二) 凌晨4:42
@.>;
<EMAIL_ADDRESS>主题: Re: [AUVSL/UWB-Jackal-World] How is the exercise effect of section 4.6 achieved (Issue #1)
Could you clarify?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Glad to hear that!


feat(json-schema): move 'null' type to @formly/core

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
Feature
What is the current behavior? (You can also link to an open issue here)
Null type for json-schema needs to be implemented manually.
What is the new behavior (if this is a feature change)?
Null type for json-schema is added to @formly/core
Please check if the PR fulfills these requirements

[X] The commit messages follow our guidelines: https://github.com/angular/angular.js/blob/master/CONTRIBUTING.md#commit-message-format
[X] A unit test has been written for this change.
[X] Running npm run build produced a successful build. (Unit testing can be done by running npm test;)
[X] My code has been linted. (npm run lint to do this. npm run build will fail if there are files not linted.)

Other information:
I think it makes most sense to add the null type to core, but open for other suggestions.
I had to get rid of the form-field wrapper, not sure if that's an issues.
@aitboudad what do you think?

Well, I'm not clearly sure about that move :thinking:, null in json-schema is used to display title and description an may also render error message for that reason we've used form-field wrapper.
I'd prefer to not hurry for null, object and array types. They should the latest part in json-schema roadmap.

That makes total sense @aitboudad - I'm closing this for now. Let's take another look at it further down the line :)


kubetail: 1.6.6 -> 1.6.7

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/kubetail/versions.
meta.description for kubetail is: '"Bash script to tail Kubernetes logs from multiple pods at the same time"'.
Release on GitHub
Compare changes on GitHub

Checks done (click to expand)

built on NixOS
/nix/store/fzf53iz6npwp4qd9s9hm9fjpfs06xwy4-kubetail-1.6.7/bin/kubetail passed the binary check.
1 of 1 passed binary check by having a zero exit code.
1 of 1 passed binary check by having the new version present in output.
found 1.6.7 with grep in /nix/store/fzf53iz6npwp4qd9s9hm9fjpfs06xwy4-kubetail-1.6.7
directory tree listing: https://gist.github.com/046741b3fca1b06039fd5f5c7f6d2c87
du listing: https://gist.github.com/b462b3462625c27ad1788deec205fa8d

Rebuild report (if merged into master) (click to expand)

4 total rebuild path(s)
1 package rebuild(s)
1 x86_64-linux rebuild(s)
1 i686-linux rebuild(s)
1 x86_64-darwin rebuild(s)
1 aarch64-linux rebuild(s)
First fifty rebuilds by attrpath
kubetail

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/fzf53iz6npwp4qd9s9hm9fjpfs06xwy4-kubetail-1.6.7 \
  --option binary-caches 'https://cache.nixos.org/ https://r-ryantm.cachix.org/' \
  --option trusted-public-keys '
  r-ryantm.cachix.org-1:gkUbLkouDAyvBdpBX0JOdIiD2/DP1ldF3Z3Y6Gqcc4c=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(r-ryantm's Cachix cache is only trusted for this store-path realization.)
Or, build yourself:
nix-build -A kubetail https://github.com/r-ryantm/nixpkgs/archive/4092605b0b12d304eef12d25638673b3f7a8b840.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/fzf53iz6npwp4qd9s9hm9fjpfs06xwy4-kubetail-1.6.7
ls -la /nix/store/fzf53iz6npwp4qd9s9hm9fjpfs06xwy4-kubetail-1.6.7/bin

cc @kalbasit for testing.

@ryantm <off-topic> do you think it's possible to have @r-ryantm request review instead of commenting on the PR? </off-topic>

@kalbasit Yes, it's possible, just a little more difficult. I'm guessing that would be preferable because OfBorg is also requesting reviews of maintainers, so it would be less notifications.


Find a way to generate SVG Sprites on Webpack level

https://github.com/SnowdogApps/magento2-alpaca-components/blob/master/gulpfile.js#L205-L216

We can use svg-sprite-loader We've used that in snowdog-website project.

But... It will require to overwrite default Storybook Webpack config, since they already have some loader for SVG.

Yeah. It's required to slice this config and add it again with required loader, as we did in Nuxt. Maybe there's another way to add, I'll check that.


Slow training time (can be fixed)

Hi Jun,
awesome work! While playing with your repo I noticed that training times are WAY slower than they should be. When using the regular nnUNetTrainer, an epoch of Hippocampus takes 22s instead of 7-8 (on RTX 4090) even though none of the Mamba stuff should be involved.
 I recommend you change the instructions to
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
(taken straight from the pytorch website. cuda 11.8 is important as it won't work with cuda 12 due to the causal conv repo)
This has the following effect for me:

regular nnUNetTrainer on Hippocampus goes from 22s -> 7.5s per epoch
nnUNetTrainerUMambaEnc goes from >60s to 24s

Note that I only verified that trainings are running, please make sure everything works fine before changing that :-)
Best,
Fabian

Hi @FabianIsensee ,
Happy New Year!
Thank you so much for the valuable comments. We will do a thorough evaluation under the new environment before making the update.
Best regards,
Jun

any update?

I am looking forward to the update. Currently it is very slow, takes 5 times more time on almost any training task (BTCV, ACDC, Synapse etc), regardless of 2d or 3d_fullres


Set up lambda for emails

This lambda takes in an email address and an html string and sends it to sendgrid.

Sengrid refused our account, so we will be using mailgun


Unit test backfill

Codecov Report

Merging #838 (9092793) into master (8ecf4aa) will increase coverage by 0.27%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #838      +/-   ##
==========================================
+ Coverage   98.69%   98.97%   +0.27%
==========================================
  Files         293      293
  Lines       15757    15757
==========================================
+ Hits        15551    15595      +44
+ Misses        206      162      -44


Tests are failing

Expected 1.1 version, but it's already 1.3 in recent MoarVM versions.

What tests are we talking about?


Truncate route description

I'm seeing a number of these:
ERRO 2024-01-13 14:26:18 Failed to decode route insert route error: Error 1406 (22001): Data too long for column 'description' at row 1

Rather than increase the VARCHAR(255) at this time, I've chosen to just truncate the description and log info about it. Truncating unicode correctly is a bit of fun.
Fixes #204

I've tested this and this fixes the DB fails I was seeing.

My len() check was also wrong. This new version has been tested now also.

I swear I left a question here to others and myself.... but I don't see it. I probably left it on another PR by accident ha, but:
I'm wondering if this is not the correct spot to do this. I was trying to preserve the original description in here with the defer, but I think maybe this is just kinda dumb.
Maybe Description should be truncated when *Route is initially formed, and just let it be truncated ?

Yeah, I think so. I've updated this.


How to convert markdown to html in realtime.

We should find some library about convert to html. and it should be easy to edit code for making slide.
or the other way is we make it. I think that is not that hard now.
ps. This issue is not about django feature. so It is not need to make by everybody.
If you wanted to implement this feature, you assign to yourselves. or ask to help to another one.

I found a js library https://github.com/evilstreak/markdown-js/blob/master/README.md
How about using this? (I did not ever use it)

We are studying Django which is a PYTHON Web framework, NOT JS one.
So I think we should use python markdown parser. ex) python-markdown2 (I've not ever use it too)
We can implement realtime thing and learn Django more, using python md parser and ajax.

Wow.... We need to talk about it. I think both of them are correct way.

We've discussed about JavaScript vs Python for parser on our team slack.
In conclusion, we decided to use JavaScript for markdown parsing.

So, we should choose a markdown-parser js library. (like above library i found)

markdown-js

Simple usage
5108 stars
Code was stopped on 2015 May

showdown

You should initiate converter on your own hands(?)
3902 stars
Mid-active developing (Aug 20)

marked

Not bad to my eyes.
10535 stars
Very-active developing (19 days ago)

Benchmark is shown on marked README doc
$ node test --bench
marked completed in 3411ms.
marked (gfm) completed in 3727ms.
marked (pedantic) completed in 3201ms.
robotskirt completed in 808ms.
showdown (reuse converter) completed in 11954ms.
showdown (new converter) completed in 17774ms.
markdown-js completed in 17191ms.

marked +1

marked! LGTM :D


how to use this library with swift 1.2?

I get compiler errors with master branch.
"Consecutive statements on a line must be separated by ';'"

Yeah because the master has been upgraded to use swift 2.0.

try with tag 2.2.0, if it's in CocoaPods do it like this:
pod 'SwiftyJSON', :git => 'https://github.com/SwiftyJSON/SwiftyJSON.git', :tag => '2.2.0'


Deleted network Brabant doesn't disappear

Someone made a hiking network relation for Noord-Brabant, while those routes should be in network relations per municipality. I deleted the network, but it is still in Knooppuntnet including the errors.

The network has been manually set to active=false in the database.
Where necessary the database views have been updated to properly exclude non-active networks, routes and nodes from reporting.
As a result, the network page still exists and the history of the network changes is preserved, but the network does not appear appear anymore in the overview page, network list or as network reference in the node and route pages.
This does not resolve the actual problem that the network deletion was not automatically detected by the analysis logic. That issue will be further handled in issue #51.


Custom collectionType binding with fluent-builder

In my binding file I apply a custom List for a specific list element via
<jaxb:property collectionType="MyList" />
When I use fluent-builder, the generarated builder code always uses ArrayList instead. I've read the source code of the plugin and it seems you're hard-coding it to use ArrayList (PluginContext.arrayListClass).
Is there anyway to change this behaviour? I could work on the source code but I would need some directions.

You're right, the ArrayList type has been hardcoded. It would certainly be possible to apply collection types. I'm looking into it...

You're right, the ArrayList type has been hardcoded. It would certainly be possible to apply collection types. I'm looking into it...

Have the same issue, seems the fluent builder didnt check a field is null or not.
If the collectionType is set each field is initialized with the collection type instance. But the fluent builder replace the collection instead of adding the values. Can be its better to check the field is null or not.
I have an problem using default values. Default values  overwritten with null.
To reproduce this behavior just create and field with default value

Class generation will create a field and initialize it
@XmlElement(required = true, defaultValue = "myValue")
protected String defValue = "myValue";
But the init Method overwrite it with null:
_product.defValue = this.defValue;
At this line this.defValue is null because withDefValue is never called for default values.
Can be an null check can solve both. But I'm not sure, is there any other possibility to use default values together with the builder api?

Have the same issue, seems the fluent builder didnt check a field is null or not.
If the collectionType is set each field is initialized with the collection type instance. But the fluent builder replace the collection instead of adding the values. Can be its better to check the field is null or not.
I have an problem using default values. Default values  overwritten with null.
To reproduce this behavior just create and field with default value

Class generation will create a field and initialize it
@XmlElement(required = true, defaultValue = "myValue")
protected String defValue = "myValue";
But the init Method overwrite it with null:
_product.defValue = this.defValue;
At this line this.defValue is null because withDefValue is never called for default values.
Can be an null check can solve both. But I'm not sure, is there any other possibility to use default values together with the builder api?

Planned for 4.2.0

Re default values: This is a different issue. Rudimentary handling of attribute default values in fluent-builder will also be added in 4.2.0


init DAQ data type update

Added signed integer types and floats.
The signal bit length is only specifiable when the data type is unsigned and floats default to 32 bits.
In addition, more signal parameters can be added such as scale, offset, minimum, maximum, and units.
The scaling and offset are applied as (x * scale) / offset whenever the message is decoded.
However, they are not automatically applied whenever you send the message.
In addition, the units are displayed in the DAQ app under the frame viewer and plot axis labels.

Is it (x * scale) / offset or (x * scale) + offset?

ah looks like its +, looks good!

@AdamBusch  my bad, it is (x * scale) + offset, this is also reflected in the can schema


Make the cookie popup less annoying

Right now it comes up all the time again, this needs to be fixed somehow.

This is already fixed


libktx needs to be updated to follow latest spec changes

KTX specification rc1 (https://github.khronos.org/KTX-Specification/, 2020-05-26) removed globalFlags from supercompression global data for BasisU.
This makes all existing KTX2 files incompatible, and accordingly requires an update to libktx to load them; currently valid (as per new spec) files fail to load with:

ERROR: Failed to transcode: File data is inconsistent with KTX spec.

Attaching a sample file which loads correctly in three.js after https://github.com/mrdoob/three.js/pull/19791
rubyrose.glb.zip

Adding @bghgary to upgrade

I've been looking at this for a bit. There are issues with the latest version of libktx.js/wasm where the GL object not exposed for some reason. Still investigating with Mark, but this might take a bit of time.

Turns out the new build in KTX-Software is causing an issue. The old build is still there, so I was able to create a build using that with Mark's help.


The game-master can choose how many steps the camels in the game need to make until the finishing line.

Closed with commit https://github.com/sopra-fs23-group-28/sopra-fs23-group-28-server/commit/436c74b63de8f3e5877b96c1cbb542101eca3c07


feat: add svelte preset

:tada: This PR is included in version 2.6.0-stage.14 :tada:
The release is available on:

npm package (@stage dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 2.6.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Add the timeout.cancel function from angular 1.5

Get from https://github.com/google/closure-compiler/blob/master/contrib/externs/angular-1.5.js#L2575-L2589

@dpurp did we leave out the reopening + $timeout.cancel on purpose?

Yes and no.  It was actually moved to angular-1.6-function-properties.js, since it didn't compile with strict checks due to an issue with Closure compiler.  It errored with properties-on-functions, wherein two functions have properties with the same name ($timeout and $interval).
The issue was fixed internally a few weeks ago, but I have been waiting on a public release of closure.  Since that happened last week, I should be merge function-properties back into the main file.  I'll look into that today.

Thanks in advance @dpurp :-)

Exported to github in: https://github.com/google/closure-compiler/commit/3a99f4ddacee6173a2d00e2803612628a79a8407

Thanks :-)


Fix API docs for Commands methods

Objective
The doc comments for Command methods are a bit inconsistent on the format, they sometimes go out of scope, and most importantly they are wrong, in the sense that they claim to perform the action described by the command, while in reality, they just push a command to perform the action.

Follow-up of #5938.
Related to #5913.

Solution

Where applicable, only stated that a Command is pushed.
Added a “See also” section for similar methods.
Added a missing “Panics” section for Commands::entity.
Removed a wrong comment about Commands::get_or_spawn returning None (It does not return an option).
Removed polluting descriptions of other items.
Misc formatting changes.

Future possibilities
Since the Command implementors (Spawn, InsertBundle, InitResource, ...) are public, I thought that it might be appropriate to describe the action of the command there instead of the method, and to add a method → command struct link to fill the gap.
If that seems too far-fetched, we may opt to make them private, if possible, or #[doc(hidden)].

@wilk10 I just added “to the queue”, since “Pushes a Command to the command queue” sounds a little too repetitive.

Tests in code should make explicit what functionality is expected. (Code is law?)

What do you mean? I'm not touching tests here.

I mean someone ought to write tests for the Commands methods to demonstrate how they ought to work.  The documentation can be incorrect/misleading, but having code which demonstrates the intention of how Commands is expected to work is worth even more than solely changing the documentation.

For example, if the Commands are actually applied at a certain Stage of the overall ECS system's operation, a set of tests could make that clear.  I have no objection to your PR.


Fix the default solution project name and add a few missing files

This produces a working build, the 0.1.0 release does not work for some reason for myself (no log created, no change to the fps decrease during simulation)

This produces a working build, the 0.1.0 release does not work for some reason for myself (no log created, no change to the fps decrease during simulation)

The 0.1.0 release was compiled as a debug build.  This causes it to link to the debug version of the MSVC 2022 runtime, which is only available if you have Visual Studio 2022 installed. Microsoft doesn't license those runtime DLLs for redistribution.

Which i obviously have installed by being able to build it :p

This produces a working build, the 0.1.0 release does not work for some reason for myself (no log created, no change to the fps decrease during simulation)

@Squall-Leonhart I've just released v0.1.1 which should be a release build. https://github.com/caspervg/sc4-disable-fps-limits/releases/tag/v0.1.1 Can you check if that one works for you?


WIP: CI: Change CI to use container image directly for lint and unit-tests

It currently looks like this is more trouble than it is worth, so i will shelve this idea for now.


README.md: Add info about PR-ing from a branch rather than from a fork

Signed-off-by: Florin Sarbu<EMAIL_ADDRESS>

@balena-ci I self-certify!


Add new course home page to the lms tests.

We have simply added a new page.  None of the older pages need to be removed.
I noticed that we also had an older task for a page that no longer exists and is not used for anything, so I have removed it.

FYI @pwnage101 on updates to new LMS tests.

I approve, feel free to merge.  Don't let this block you, but I'm curious how you got to that task ratio of 10?

@pwnage101 I picked 10 because this is replacing the existing courseware tab, and the courseware tasks were set to 10 as well.


Make line number and column number available in full sreen (ctrl+k+z)

When VS code runs in Zen mode, it is the most effective (provided you know shortcuts). But one thing I miss is glancing down at the number of columns a line is occupying. This is especially needed if PEP 8 is to be followed in python.

Please use this setting zenMode.hideLineNumers: false


FR: Toggle thumbs up with radiant in the background

It would be nice to be able to toggle thumbs up while the radiant player resides in the background.
In the same way as the apple play/pause, next and previous buttons work.

CMD+J is the shortkey combination for toggling thumbs up.

Thumbs up to this

:+1: to your thumbs up


edit websocket.rst prose

a bit more terse and hopefully a bit more clear English (from my non-native speaker's point of view)

cherry-picked, thanks a lot!


Feature/45 data validation

I think this PR is ok with the tests but I would be more comfortable to merge it if we also had the rake tasks to normalise existing data for prod...what do you think?

I agree, if you are happy with the PR then we'll just leave it open, and I'll start work on the data identification scripts.


feat: Add OnStopServer / OnStopClient methods to Network Authenticator

Not having these was an oversight.  When StopServer / StopClient are called there's no cleanup, so if they're started again, you'll get warnings on attempting to replace the handlers since they weren't removed.
Additionally, if users want to implement a Cancel feature to authentication, they should call StopClient and there's no callback to the authenticator to break out of the authentication without these additions.

can you elaborate on why this is necessary please?

can you elaborate on why this is necessary please?

Description added to the PR


[Cypress] - Split the test

The main scenario lasts 1 hour, that's a bit longer.
https://github.com/epinio/epinio-end-to-end-tests/actions/runs/2514148468
We would like to try splitting it in smaller / faster tests.
Maybe a specific test for each of these steps:

Menu
Applications with and without s3 / external registry
Configurations
Services
Namespaces

Fixed!
https://github.com/epinio/epinio-end-to-end-tests/actions/runs/2545470372


file: 5.32 -> 5.33

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools.
This update was made based on information from https://repology.org/metapackage/file/versions.
These checks were done:

built on NixOS
ran ‘/nix/store/n5afz3w8g0mj9pf5qcndb3kdz4jbljca-file-5.33/bin/file --help’ got 0 exit code
ran ‘/nix/store/n5afz3w8g0mj9pf5qcndb3kdz4jbljca-file-5.33/bin/file help’ got 0 exit code
found 5.33 with grep in /nix/store/n5afz3w8g0mj9pf5qcndb3kdz4jbljca-file-5.33
directory tree listing: https://gist.github.com/b6f0a44bc22e88f13fe5dbc4f56d9296

Success on x86_64-darwin (full log)
Attempted: file
Partial log (click to expand)
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/private/tmp/nix-build-file-5.33.drv-0/file-5.33'
make[1]: Leaving directory '/private/tmp/nix-build-file-5.33.drv-0/file-5.33'
post-installation fixup
gzipping man pages under /nix/store/inb4p6wxx717xg6ixjwli46z12x3w29p-file-5.33/share/man/
strip is /nix/store/kdff2gim6417493yha769kh00n63lnrw-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/inb4p6wxx717xg6ixjwli46z12x3w29p-file-5.33/lib  /nix/store/inb4p6wxx717xg6ixjwli46z12x3w29p-file-5.33/bin
patching script interpreter paths in /nix/store/inb4p6wxx717xg6ixjwli46z12x3w29p-file-5.33
/nix/store/inb4p6wxx717xg6ixjwli46z12x3w29p-file-5.33

Success on x86_64-linux (full log)
Attempted: file
Partial log (click to expand)
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/n5afz3w8g0mj9pf5qcndb3kdz4jbljca-file-5.33
shrinking /nix/store/n5afz3w8g0mj9pf5qcndb3kdz4jbljca-file-5.33/bin/file
shrinking /nix/store/n5afz3w8g0mj9pf5qcndb3kdz4jbljca-file-5.33/lib/libmagic.so.1.0.0
gzipping man pages under /nix/store/n5afz3w8g0mj9pf5qcndb3kdz4jbljca-file-5.33/share/man/
strip is /nix/store/j75dgadrff2d1fyc4fczmcgqkid2imdx-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/n5afz3w8g0mj9pf5qcndb3kdz4jbljca-file-5.33/lib  /nix/store/n5afz3w8g0mj9pf5qcndb3kdz4jbljca-file-5.33/bin
patching script interpreter paths in /nix/store/n5afz3w8g0mj9pf5qcndb3kdz4jbljca-file-5.33
checking for references to /build in /nix/store/n5afz3w8g0mj9pf5qcndb3kdz4jbljca-file-5.33...
/nix/store/n5afz3w8g0mj9pf5qcndb3kdz4jbljca-file-5.33

Success on aarch64-linux (full log)
Attempted: file
Partial log (click to expand)
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/7c3aqy7r6d7fwar6j115g1vl91yxypv2-file-5.33
shrinking /nix/store/7c3aqy7r6d7fwar6j115g1vl91yxypv2-file-5.33/bin/file
shrinking /nix/store/7c3aqy7r6d7fwar6j115g1vl91yxypv2-file-5.33/lib/libmagic.so.1.0.0
gzipping man pages under /nix/store/7c3aqy7r6d7fwar6j115g1vl91yxypv2-file-5.33/share/man/
strip is /nix/store/j7d4mr0ikv974ig7yzhknpsq288js4bs-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/7c3aqy7r6d7fwar6j115g1vl91yxypv2-file-5.33/lib  /nix/store/7c3aqy7r6d7fwar6j115g1vl91yxypv2-file-5.33/bin
patching script interpreter paths in /nix/store/7c3aqy7r6d7fwar6j115g1vl91yxypv2-file-5.33
checking for references to /build in /nix/store/7c3aqy7r6d7fwar6j115g1vl91yxypv2-file-5.33...
/nix/store/7c3aqy7r6d7fwar6j115g1vl91yxypv2-file-5.33


Encoding problem with some Unicode characters

When I try to process a video with some Czech accented characters (such as "ě", "š", "ž" etc.), the file gets written to the file system with the correct name, but FFmpeg reads the characters incorrectly.
Here's the FS and fferr output with a file named "Příliš žluťoučký kůň úpěl ďábelské ódy.mxf":
[info] run FS.writeFile Příliš žluťoučký kůň úpěl ďábelské ódy.mxf <165155412 bytes binary file>
...
[fferr] PY�a ~lueouk��H 򰿬 ?ᢥlsk頳dy.mxf: No such file or directory
It seems like FFmpeg itself expects some different encoding.

It also seems like the non-breaking space character (\u00a0) is causing problems for the FS.

Same issue happens with Korean too :(

Same issue with chinese characters.

same issue with audio files metadata.
any luck solving it?
must be a way to make it work with utf8
maybe writing custom config file and setting some thngs....


cmd/compile: Inefficiencies in binary code

What version of Go are you using (go version)?

$ go version
go version go1.20.3 linux/amd64

Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/czhong4/.cache/go-build"
GOENV="/home/czhong4/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/czhong4/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/czhong4/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/czhong4/gocode/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/czhong4/gocode/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/czhong4/gocode/go/src/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build822587075=/tmp/go-build -gno-record-gcc-switches"

What did you do?
I ran a redundant loads detecting tool on function strconv.FormatFloat().

What did you expect to see?
Redundant loads can be optimized by the compiler.
What did you see instead?

In function rightShift() in package strconv,

...
for ; n>>k == 0; r++ {
        if r >= a.nd {
...

a.nd is a loop invariant. But the compiler failed to put it in a register, resulting in a.nd being loaded from memory repeatedly in each iteration.

In function Assign() in package strconv,

...
a.nd = 0
for n--; n >= 0; n-- {
	a.d[a.nd] = buf[n]
	a.nd++
}
...

a.nd is loaded from memory twice in a single iteration.

CC @golang/compiler
CC @dr2chase in particular, who I think was recently looking into loop invariant hoisting. I think there's an open bug for loop invariant hoisting already, and IIUC (though I could be wrong) that optimization doesn't make a meaningful difference in real-world programs, only microbenchmarks. It might still be worth doing as it adds up, but I just wanted to add that context.

a.nd = 0
for n--; n >= 0; n-- {
	a.d[a.nd] = buf[n]
	a.nd++
}

Hoisting the a.nd loads is tricky because the compiler currently doesn't do any alias analysis, so the write to a.d could be one of the bytes of a.nd. This is on the pretty easy end of alias analysis, though, as the same base pointer makes it pretty obvious. So maybe doable.
Some loop rotation might get you down to 1 load of a.nd per iteration.


Model in Keras or ONNX format?

Hi! This looks really promising and interesting, good work! Would it be possible to get the actual trained model either in Keras format (.h5) or in ONNX? I'd like to use the model in MATLAB, but I can only import either Keras or ONNX formats.

Good point. Right now, only tensorflow format is supported, as this was the framework that I used for training.
I'm planning to release the training scripts soon, so that anyone can replicate the architecture/training strategy with another framework!

I was able to convert the face detector to ONNX. I made a fork with an example usage for the C# ONNX Runtime: https://github.com/brianpopow/faced
The conversion was done like this:
python -m tf2onnx.convert --verbose --input face_yolo.pb --opset 15 --inputs img:0 --use_default img --outputs prob:0,x_center:0,y_center:0,w:0,h:0 --output faced.onnx

Unfortunately converting the face corrector network to ONNX did not work as expected. The produced outputs does not seem to match the original network.
It still works reasonable good without the corrector in my opinion and is also fast. On my CPU it takes 40 to 45 ms to detect the faces and that includes loading the image and resize it to 288x288.
Here is an example:


docker-Elk swarm to kubernetes

Hello,
How can i deploy the exact same stack to kubernetes rather than swarm ?
Thank you in advance!

You could use the official Helm chart: https://github.com/helm/charts/tree/master/stable/elastic-stack
Closing because off topic. Feel free to ask more details in the Kubernetes community.


LiveFyre comment: Can source code transformation be avoided?

From https://docs.microsoft.com/en-us/nuget/create-packages/source-and-config-file-transformations#comments-container:
Source Code transformations (MyClass.cs.pp) seem to be applied during upgrade. I'm trying to include a sample class file, but I don't want it to be added back to the project every time the end-user updates the package. Is there a way around this?

Rolling up to #779


Add a display value for phone numbers

The formatters used for ToString() are based upon international standard formats and do not necessarily match the user expectation of how a number should be displayed.
For example a French phone number would usually be shown as XX XX XX XX which doesn’t match any international formats.

This will be implemented and used to improve the ToString(“E.123”), ToString(“N”) and ToString(“RFC3966”) formats:
E.123
Current outputs: +44 114 2726444, +1<PHONE_NUMBER>, +33 140477283
Future outputs: +44 114 272 6444, +1<PHONE_NUMBER>, +33 1 40 47 72 83
N
Current outputs: (0114) 2726444,<PHONE_NUMBER>,<PHONE_NUMBER>
Future outputs: (0114) 272 6444,<PHONE_NUMBER>, 01 40 47 72 83
RFC3966
Current outputs: tel:+44-114-2726444, tel:+1-212-4841200, tel:+33-140477283
Future outputs: tel:+44-114-272-6444, tel:+1-212-484-1200, tel:+33-1-40-47-72-83


Init command is not compatible with volta

Describe the bug
Volta is a node version manager like nvm, and camouflage init is not compatible with volta library structure.
In volta, npm root -g result is /Users/bob/.volta/tools/image/node/16.13.0/lib/node_modules, but camouflage is located in /Users/bob/.volta/tools/image/packages/camouflage-server.
From what I understand, site_root should be located as bin/camouflage.js parent directory.
let site_root = path.join(__dirname, ".."); should be good enough in many cases.
To Reproduce
Steps to reproduce the behavior:
$ volta install camouflage-server
$ camouflage init
node:internal/fs/utils:344
    throw err;
    ^

Error: ENOENT: no such file or directory, stat '/Users/bob/.config/yarn/global/node_modules/camouflage-server/mocks'
    at Object.statSync (node:fs:1536:3)
    at Object.statSync (/Users/bob/.volta/tools/image/packages/camouflage-server/lib/node_modules/camouflage-server/node_modules/graceful-fs/polyfills.js:311:34)
    at statSync (/Users/bob/.volta/tools/image/packages/camouflage-server/lib/node_modules/camouflage-server/node_modules/fs-extra/lib/util/stat.js:10:52)
    at getStatsSync (/Users/bob/.volta/tools/image/packages/camouflage-server/lib/node_modules/camouflage-server/node_modules/fs-extra/lib/util/stat.js:24:19)
    at Object.checkPathsSync (/Users/bob/.volta/tools/image/packages/camouflage-server/lib/node_modules/camouflage-server/node_modules/fs-extra/lib/util/stat.js:49:33)
    at Object.copySync (/Users/bob/.volta/tools/image/packages/camouflage-server/lib/node_modules/camouflage-server/node_modules/fs-extra/lib/copy-sync/copy-sync.js:24:38)
    at Object.<anonymous> (/Users/bob/.volta/tools/image/packages/camouflage-server/lib/node_modules/camouflage-server/bin/camouflage.js:88:9)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32) {
  errno: -2,
  syscall: 'stat',
  code: 'ENOENT',
  path: '/Users/bob/.config/yarn/global/node_modules/camouflage-server/mocks'
}

Expected behavior
Empty directory is initialized.
Desktop (please complete the following information):

OS: [e.g. iOS] macOS
Browser [e.g. chrome, safari] N/A
Version [e.g. 22] 12.1

There was a similar issue with yarn earlier, #139
Seems appropriate to use degit for initialising a Camouflage project. I have created a template repository for such use cases. Please use following command to replace camouflage init
degit camouflagejs/init myproject

OR
 npx degit camouflagejs/init myproject

TODO: Update docs


I need to do hot reload every time I run my app on Emulator to get the correct size of Widgets

Here Is The Code :
   class S2CSE_EnC extends StatelessWidget {
  const S2CSE_EnC({super.key});

 @override
 Widget build(BuildContext context) {
 // test

final scaleFactor = MediaQuery.of(context).textScaleFactor;
double devH = MediaQuery.of(context).size.height;
double devW = MediaQuery.of(context).size.width;
double x;
num y;
if (devW <= 375) {
  x = 0.18;
  y = 17;
} else {
  x = 0.19;
  y = 18;
}

return SafeArea(
  child:
      Column(mainAxisAlignment: MainAxisAlignment.spaceAround, children: [
    SizedBox(
      height: devH > 1000 ? 3.h : 2.h,
    ),
    Container(
      padding: EdgeInsets.all(6),
      decoration: BoxDecoration(
        color: Colors.white60,
        borderRadius: BorderRadius.all(
          Radius.circular(12),
        ),
      ),
      child: Padding(
        padding: EdgeInsets.all(8),
        child: Text(
          "Semester II",
          style: GoogleFonts.openSans(
            fontSize: devH > 1000 ? 57 * scaleFactor : 23.sp,
            color: Colors.black,
            fontWeight: FontWeight.w600,
          ),
        ),
      ),
    ),

Hi @hacker0id
The goal of this Github repo is not to give assistance on personal code or the whole package/plugin project,
but to improve flutter reporting bugs or feature requests.
If you encounter behavior that you believe could be a bug, please provide a completed and minimal reproducible code sample that represents the problem. Also, please provide the output of flutter doctor -v as well.
Instead, if you think that the issue might depend on your implementation, please see flutter's community for resources and ask questions like this. You may also get some help if you post it on Stack Overflow and flutterhelp.
Thank you!


Different behavior between eager execution and tf.function

Hello Tensorflow community
I have a loss function with a few logarithms in it:
def NegLogProb(a, b, targ):
    return -tf.reduce_mean(tf.reduce_sum(tf.math.log(a*b) + (a-1)*tf.math.log(targ+1e-8) + (b-1)*tf.math.log((1-targ**a)+1e-8), axis=-1))

This is a form of the beta distribution. My network outputs parameters a and b: the range for both outputs is >=(1+1e-8). targ is a spectrum whose values are normalized from 0-1. There are three terms in this equation. Where necessary you can see that I added 1e-8 to avoid log(0). I believe I have put in all the safeguards to prevent the logarithm from failing.
This equation works when running in eager execution, but if I create a graph out of it, the equation will always output inf followed by NaN. Can anyone account for the fact that the behavior changes when creating a graph?
I am using Tensorflow version 2.1.0
Many thanks in advance.

@jlapin1
The code shared above is not sufficient to reproduce the issue reported, please share code such that we can reproduce your issue or is possible share a colab gist with the error reported.

Hopefully this is enough code to help you reproduce it. If it helps, targ, a, b all have dimensions batch_size, 9100
def NegLogProb(a, b, targ):
    return -tf.reduce_mean(tf.reduce_sum(tf.math.log(a*b) + (a-1)*tf.math.log(targ+1e-8) + (b-1)*tf.math.log((1-targ**a)+1e-8), axis=-1))

LossFunc = NegLogProb
def train_step(samples, targ):
    with tf.GradientTape() as tape:
        a,b = model(samples, training=True)
        loss = LossFunc(a, b, targ)
    grads = tape.gradient(loss, model.trainable_variables)
    opt.apply_gradients(zip(grads, model.trainable_variables))
    return loss, (a,b)

simintime = []
testintime = []
def train(epochs, batch_size=100, svwts=False):
    graph = train_step#tf.function(train_step)

    tot = intf.shape[0]
    steps = (tot//batch_size) if tot%batch_size==0 else (tot//batch_size)+1

    for m in range(epochs):
        tots = 0
        start_epoch = time()
        for n in range(steps):
            start_step = time()
            begin = n*batch_size
            end = (n+1)*(batch_size)

            targ = create_target(Pos[begin:end])
            sim, var = graph(intf[begin:end], targ)

            tots+=sim.numpy()*(end-begin)
            sys.stdout.write("\rStep %d/%d; -log_prob: %8.1f; Time: %.1f s"%(n+1, steps, sim, time()-start_step))
        final_sim = tots/tot
        simintime.append(final_sim)

I ran the above code on tf 2.3, tf 2.4 and nightly and do not see nay output or error, please find the gist here, can you please share a colab gist with the error reported.

My apologies Saduf. The code as I sent it to you is running with eager execution, since the tf.function is commented out (right below the beginning of def train()).
Can you run it again but replace the train subroutine with this code? Notice that the variable "graph" is now an actual tensorflow graph of the function "train_step"
def train(epochs, batch_size=100, svwts=False):
    graph = tf.function(train_step)

    tot = intf.shape[0]
    steps = (tot//batch_size) if tot%batch_size==0 else (tot//batch_size)+1

    for m in range(epochs):
        tots = 0
        start_epoch = time()
        for n in range(steps):
            start_step = time()
            begin = n*batch_size
            end = (n+1)*(batch_size)

            targ = create_target(Pos[begin:end])
            sim, var = graph(intf[begin:end], targ)

            tots+=sim.numpy()*(end-begin)
            sys.stdout.write("\rStep %d/%d; -log_prob: %8.1f; Time: %.1f s"%(n+1, steps, sim, time()-start_step))
        final_sim = tots/tot
        simintime.append(final_sim)

@jlapin1
Please share a colab gist with the error reported for us to analyse.

I'm not 100% sure how to share the colab gist, but I just ran this in colab and this is what I got. No error statement but clearly there is different behavior between eager execution and the tf graph
import numpy as np
import os
# import re
import sys
import tensorflow as tf
import tensorflow.keras.layers as L
import tensorflow.keras.activations as A
from time import time
import matplotlib.pyplot as plt
plt.close('all')
from matplotlib.ticker import MultipleLocator

targ = tf.constant(
    [[0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.20933868,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.2082187 , 0.        ,
        0.18404533, 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.37072948, 0.        ,
        0.        , 0.        , 0.        , 0.31289002, 0.        ,
        0.        , 0.        , 0.915976  , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.23793682, 0.        , 0.        ,
        0.        , 0.4674082 , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.18847302,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.2036586 , 0.17317393, 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.29346868,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.40215385, 0.        , 0.16911463, 0.        ,
        0.        , 0.        , 0.24255644, 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.24563654, 0.        , 0.        , 0.        , 0.33256173,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.30373514, 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.2429015 , 0.15512827, 0.        , 0.        ,
        0.        , 0.41071343, 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.29341364, 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.21696085, 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.1478434 , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.35621345, 0.        , 0.        , 0.        ,
        0.        , 0.15217698, 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.38775736, 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.19092496,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.18351643, 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.5620028 ,
        0.3221243 , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.18488021, 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.14928055, 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.28359178,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 1.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.17243865, 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.4688842 , 0.30614266, 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.18065585, 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.1995775 , 0.        , 0.        , 0.        , 0.        ,
        0.59800917, 0.        , 0.        , 0.        , 0.        ,
        0.2842905 , 0.        , 0.        , 0.98819506, 0.20961763,
        0.        , 0.        , 0.        , 0.        , 0.30928153,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.        , 0.        ]], dtype=tf.float32
)

sample = tf.constant([[19,  4, 15, 15,  4, 16, 16,  6,  8,  8, 20, 21, 21, 21, 21, 21,
        21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21,
        21, 21, 21, 21, 21, 21, 21, 21, 21]], dtype=tf.int32)

class ResBlock2(L.Layer):
    def __init__(self, filters, KS=3, conv_type=L.Conv1D, act=A.relu):
        super(ResBlock2, self).__init__()
        self.filters = filters
        self.conv_type = conv_type
        self.norm1 = L.BatchNormalization()
        self.norm2 = L.BatchNormalization()
        self.conv1 = conv_type(filters, KS, 1, padding='SAME', use_bias=False) # bias removed after running
        self.conv2 = conv_type(filters, KS, 1, padding='SAME', use_bias=False)
        self.act = act
    def build(self, x):
        self.shortcut = self.conv_type(self.filters, 1, 1) if self.filters!= x[-1] else A.linear
    def call(self, x):
        out = self.act(self.norm1(self.conv1(x)))
        out = self.act(L.Add()([self.shortcut(x), self.norm2(self.conv2(out))]))
        return out

def CustomBlock(feat_map, rang, feat_out=128, residual=True, pad='SAME', alpha=0.0):
    rang = tf.range(*rang)
    out = tf.concat([L.Conv1D(feat_out, int(m), 1, padding=pad)(feat_map) for m in rang], axis=-1)
    out = L.BatchNormalization()(out)
    if residual:
        shortcut = L.Conv1D(out.shape[-1], 1, 1) if feat_map.shape[-1]!=out.shape[-1] else A.linear
        out = L.Add()([shortcut(feat_map), out])
    return A.relu(out, alpha)

upsample = lambda x: tf.reshape(tf.tile(tf.expand_dims(x, axis=2), [1,1,2,1]), (-1, 2*x.shape[1], x.shape[-1]))

def Model_beta(seq_len,
               AA_types,
               out_dim,
               floors=(1,1),
               filtfirst=64,
               rang=(2,10,1),
               outin=(3,5),
               filtmid=(150, 200),
               filtlast=200):
    inp = L.Input((seq_len,), dtype=tf.int32)

    out = tf.one_hot(inp, AA_types) # Original
    out = CustomBlock(out, (3,4,1), feat_out=64, residual=False, alpha=0.0)

    outer,inner = outin
    filts = np.linspace(filtmid[0], filtmid[1], outer, dtype='int')
    for m in range(outer):
        out = upsample(out) if m>0 else out
        for n in range(inner):
            filters = filts[m]
            ks = 3
            out = ResBlock2(filters, ks)(out)

    out = CustomBlock(out, (1,2,1), filtlast, residual=False)

    a = L.Conv1D(out_dim, 1, 1, activation='relu')(out)
    a = L.GlobalAveragePooling1D()(a)
    a = tf.squeeze(a)+floors[0]
    b = L.Conv1D(out_dim, 1, 1, activation='linear')(out)
    b = L.GlobalAveragePooling1D()(b)
    b = tf.squeeze(a)+floors[1]

    return tf.keras.Model(inputs=inp, outputs=[a,b])

model = Model_beta(41, 22, 910, 2*(1+1e-6,))
opt = tf.keras.optimizers.Adam(3e-4)

def NegLogProb(a, b, targ):
    return -tf.reduce_mean(tf.reduce_sum(tf.math.log(a*b) + (a-1)*tf.math.log(targ+1e-8) + (b-1)*tf.math.log((1-targ**a)+1e-8), axis=-1))

def train_step(samples, targ):
    with tf.GradientTape() as tape:
        a,b = model(samples, training=True)
        loss = NegLogProb(a, b, targ)
    grads = tape.gradient(loss, model.trainable_variables)
    opt.apply_gradients(zip(grads, model.trainable_variables))
    return loss, (a,b)

outeager,_ = train_step(sample, targ)

print(outeager)

graph = tf.function(train_step)

outgraph,_ = graph(sample, targ)

print(outgraph)

And the output is
tf.Tensor(1576.0958, shape=(), dtype=float32)
tf.Tensor(inf, shape=(), dtype=float32)

This is a bug in the constant folding logic (which is not applied in eager execution). See example below.
In a very basic test, increasing the epsilon from 1e-8 to 1e-7 was enough to avoid the inconsistency, not sure if that's acceptable for your model.
import tensorflow as tf

tf.config.optimizer.set_experimental_options(
    {'constant_folding': True})  # Returns correct results with False

@tf.function
def NegLogProb(a, b, targ):
    return tf.add((1 - tf.pow(targ, a)), 1e-7)

NegLogProb(tf.constant([1.0]), tf.constant([1.0]), tf.constant(1.0))

Alternatively, you may be able to replace x + eps with tf.where(x > 0.0, x, 1e-8), which is more accurate for very small values as well.

Thank you mdanatg. That should fix my issue because 1e-8 was an arbitrary choice, 1e-7 should be sufficiently small as well.

Was able to replicate the issue in TF 2.6.0-dev20210531,please find the gist here..Thanks !

Could you please take a look at this gist and confirm whether the issue still persists on tf-nightly. Thank you!

Hi,
I tried to execute the alternative approach and it was executed without any error. Kindly find the gist of it here.
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.


clean.sh does not exist

README.md refers to clean.sh used in case of failed installations. This script does not exist.
Unclear whether clean-up.sh is meant here or whether this particular script is used to perform some post install tasks and should not be used in case of failed installations. Please clarify the doc.

Thanks, it is clean-up.sh

Fixed readme. Will merge soon.


stop the auto load

PhoneGap CLI, node & npm versions
Expected Behaviour
Actual Behaviour
Steps to Reproduce
Sample Code that illustrates the problem
Error messages

there is no option to turn off the auto load with the gui

Please follow the Issue Template to explain this issue clearly.

omg what do  you not understand!?!?!
"there is no option to turn off the auto load with the gui"
@maverickmishra read read read

@crisium This is a command line tool, there is no gui ... explain explain explain ...
You probably meant the desktop app.
This has been on the backlog for a long time, so it is doubtful to happen soon, but you could go and post a comment there.
https://github.com/phonegap/phonegap-app-desktop/issues/255


Changing meta description to boost CTR

Trello card: https://trello.com/c/2CruTKgi/1743-content-review-meta-descriptions-for-pages
Editing meta descriptions to increase click through rates.

@Tomcowan1 - I've rebased this, ready to merge once approved


[Release] sdk/resourcemanager/hybridaks/armhybridaks/0.1.0

https://github.com/Azure/sdk-release-request/issues/3150

/azp run prepare-pipelines

/azp run go - armhybridaks - ci


Completed all formatting and additions except adding actual content.

This adds a bunch of formatting stuff and a cool search bar feature that has taken me far to long to complete.

Totally agree with the comment about dropdown. However I worked way to hard on the silly little search bar to let it go! :-) Plus no time to change, will remember for future.


Missing Paramater Password at Build Time

Hi Team,
I am having issues building my next app inside a docker file with iron session.
I have created a wrapper function around withIronSession so I can use with api or getServerSideProps without having to duplicate all the withIronSession options everytime I use it.
export type NextIronRequest = NextApiRequest & { session: Session }
export type NextIronServerSideProps =
  | GetServerSidePropsContext & { req: NextIronRequest }

export const secureEndpoint = (handler) =>
  withIronSession(handler, {
    password: process.env.SESSION_SECRET,
    cookieName: 'secure_session',
    cookieOptions: {
      secure: process.env.NODE_ENV === 'production',
      sameSite: true,
      expires: dayjs().add(1, 'day').toDate(),
    },
  })

Use case as so:
export const getServerSideProps = secureEndpoint(
  async (ctx: NextIronServerSideProps) => {
    const user = ctx.req.session?.get('user')

    if (!user) {
      return {
        redirect: {
          destination: '/login',
        },
      }
    }

    return { props: { user } }
  }
)

OR -

const handler = async (req: NextIronRequest, res: NextApiResponse) => {
  if (!req.session?.get('user')) {
    res.status(401)
    return res.send('Unauthorised')
  }

  if (req.method === 'POST') return ADD_NEW_BOOKING(req, res)
  if (req.method === 'GET') return GET_BOOKINGS(req, res)

  res.status(404)
  return res.send('')
}

export default secureEndpoint(handler)

It's working perfectly in dev, I can run npm run build no issues as well, but as soon as I try to run npm run build inside a Docker container for deployment, I am hitting the error:
#12 30.83 Error: next-iron-session: Missing parameter `password`
#12 30.83     at throwOnNoPassword (/app/node_modules/next-iron-session/dist/index.js:26:9)
#12 30.83     at withIronSession (/app/node_modules/next-iron-session/dist/index.js:107:14)
#12 30.83     at secureEndpoint (/app/.next/server/pages/login.js:78:97)
#12 30.83     at Module.1368 (/app/.next/server/pages/login.js:41:104)
#12 30.83     at __webpack_require__ (/app/.next/server/webpack-runtime.js:25:43)
#12 30.83     at __webpack_exec__ (/app/.next/server/pages/login.js:143:52)
#12 30.83     at /app/.next/server/pages/login.js:144:79
#12 30.83     at Function.__webpack_require__.X (/app/.next/server/webpack-runtime.js:108:21)
#12 30.83     at /app/.next/server/pages/login.js:144:47
#12 30.83     at Object.<anonymous> (/app/.next/server/pages/login.js:147:3) {
#12 30.83   type: 'Error'
#12 30.83 }

It appears to me that the build script is trying to interpolate the process.env at build time instead of run time, because if I hard code in the password it builds fine inside docker.
Any ideas why this would be happening? Or has anyone successfully deployed inside a docker container without having to pass the password as an ARG / ENV inside the Dockerfile? README clearly says 'The password is a private key you must pass at runtime'
TIA
EDIT: Removing my password from my local .env file and attempting to build locally also fails with same error. Defiitely trying to get the password at build time not runtime.

Hey @joelrozen I believe this is more of a Next.js generic question on how to ensure env variables are only filled at runtime rather than at builtime.
If you remove next-iron-session form the question you'll have the same issue. By default Next.js will try to replace all process.env.* variables when you run npm run build.
I agree that the README of this lib is misleading on this question though.
Could you check how to NOT replace env variables at build time and only at runtime using Next.js? We don't do anything particular in the lib on that matter.
Thanks!

@vvo I don't believe this is a generic next.js question, as I have built tens of docker images of nextjs projects all of which contain env vars (that don't exist at build time) without issue.
The compiled code in dist also still references process.env.* anyway, as it should. I believe in this case therre might be a prebuild script happening somewhere that is attempting to run iron-session and iron-session is throwing the error. I have managed a temporary solution as follows:
export const secureEndpoint = (handler) =>
  withIronSession(handler, {
    password: process.env.SESSION_SECRET || 'a',
    cookieName: 'secure_session',
    cookieOptions: {
      secure: process.env.NODE_ENV === 'production',
      sameSite: true,
      expires: dayjs().add(1, 'day').toDate(),
    },
  })

Which when built comes out exactly the same, so in prod it will still reference the SESSION_SECRET instead of 'a' but this isn't an ideal solution imo.

Hey @joelrozen thanks for digging this in. There's no prebuild script or anything that could trigger this behavior in the source code of next-iron-session.
While counterintuitive, I believe this is the current normal behavior of Next.js.
This issue might be one where you could read/ask for more details: https://github.com/vercel/next.js/issues/22266
And also from their docs

https://nextjs.org/docs/basic-features/environment-variables
The way we use next iron session might be the issue here, when we do "export const getServerSideProps = secureEndpoint(" then at build time for a page Next.js will try to run this code (even if not included in the static build, since this is not part of the static page). This is why you're seeing this error and I would love to learn more if you jump in the linked Next.js issue.
Thanks!

As I understand it, export const getServerSideProps = withIronSession(...) needs to be evaluated at build time because getServerSideProps needs to be assigned a value when loading that module. The callback is never actually called during pre-rendering afaik and the password is not actually required to set up the handler.
Would it make sense to move the password requirement to run-time? (Instead of in the function argument destructuring as it is now)

@vvo I agree with @aphofstede and @joelrozen that this is indeed a next-iron-session issue, as I use multiple other empty env variables that don't get evaluated at build time, and this does cause an issue with docker. I have not run into this issue besides with this library, and I think what @aphofstede would probably be an ideal solution

What I meant in my earlier comment was to remove the throwOnNoPassword() checks and do them inside the callback instead. (which would cause those checks to only run at run-time)
Something like:
export function withIronSession(
  withIronSessionWrapperHandler,
  {
    ttl = 15 * 24 * 3600,
    cookieName,
    password,
    cookieOptions = {},
  },
) {
  return async function withIronSessionHandler(...args) {
    const handlerType = args[0] && args[1] ? "api" : "ssr";
    const req = handlerType === "api" ? args[0] : args[0].req;
    const res = handlerType === "api" ? args[1] : args[0].res;

    const options = {
      ttl,
      cookieName: cookieName || throwOnNoCookieName(),
      password: password || throwOnNoPassword(),
      cookieOptions
    }
    await applySession(req, res, options);
...

If an alternate API would work better with typescript we could get behind that, though I think this issue mainly comes from the withIronSession() callback pattern. When defining server side props statically I think one could already do something along these lines:
export const getServerSideProps = async ({req, res}) => {
    const session = await applySession(req, res, { cookieName, password });
    ...
}

Hey there, I believe the new version (v6) now does what you expect, you can check it here: https://github.com/vvo/iron-session/releases/tag/v6.0.0

I also ran into this issue, I renamed my .env file from .env.development to .env and fixed the issue. Leaving this here incase it a fix for anyone


DO NOT MERGE Tests Seraph17 master

tests for https://github.com/material-components/material-components-web-react/pull/633

Codecov Report

Merging #672 into rc0.10.0 will increase coverage by 0.06%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##           rc0.10.0     #672      +/-   ##
============================================
+ Coverage     94.81%   94.87%   +0.06%
============================================
  Files            68       68
  Lines          2852     2831      -21
  Branches        431      426       -5
============================================
- Hits           2704     2686      -18
+ Misses           51       50       -1
+ Partials         97       95       -2

Impacted Files
Coverage Δ

packages/tab-bar/index.tsx
85.71% <100%> (ø)
:arrow_up:

packages/fab/index.tsx
92% <0%> (-0.31%)
:arrow_down:

packages/chips/Chip.tsx
97.77% <0%> (-0.21%)
:arrow_down:

packages/chips/ChipSet.tsx
98.64% <0%> (-0.15%)
:arrow_down:

packages/ripple/index.tsx
87.82% <0%> (+2.35%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9b66812...5386011. Read the comment docs.


Approach for nesting a map

Related to https://github.com/jkrumbiegel/Chain.jl/issues/42#issuecomment-988614614 & https://github.com/jkrumbiegel/Chain.jl/issues/44
Is there a recommended approach for nesting a map? This would allow arbitrary nesting in a single point-free expression — something that's really difficult in languages I've explored.
For example, if we wanted to use @chain for the whole expression, it would currently look something like:
@chain "a|x\nb| y z " begin
    strip
    split("\n")
    map(_) do x
        @chain x begin
            split("|")
            map(_) do y
                @chain y strip split
            end
        end
    end
end

But the x and y aren't needed here! Assuming there isn't an existing recommended approach — could we extent Chain to handle those too, with feature for reducing map(_) do x; @chain x begin down to something like map @chain do, so we could have:
@chain "a|x\nb| y z " begin
    strip
    split("\n")
    map @chain do
        split("|")
        map @chain do
            strip
            split
        end
    end
end

I'm not sure of the exact form that's both concise and idiomatic — I don't have a view on the exact syntax. And feel free to say this is a bad idea... Thanks again.

I was thinking along similar lines, I've been testing out things in this branch https://github.com/jkrumbiegel/Chain.jl/tree/function-chain
Your proposed syntax doesn't work, you can always test such things by prefacing the code with Meta.@dump which will give you the syntax tree (if it's parseable, which this is not) even if nothing in it corresponds to real variables and functions.
The do syntax needs a call-like object before it (with parentheses) so I thought about this:
@chain "a|x\nb| y z " begin
    strip
    split("\n")
    @chain map() do
        split("|")
        @chain map() do
            strip
            split
        end
    end
end

This would then work for arbitrary functions like filter, etc, that take a closure as the first argument which itself expects one argument.

That looks really good. I'd be more than happy to test it over the next few days.

I even thought about transforming any do syntax without a closure argument, even without @chain in front, but that's probably a step too far.

Your sense is obv better than mine — but I don't think it'd be possible to have ambiguity, given that map() do \n isn't otherwise valid. Someone could write map() do x \n if they wanted the standard do syntax. (But OTOH I guess easier to expand than retract later...)

Yes for map that's true, but I'd want it to work for any function that takes a closure, and some don't have any args and that would probably be a really confusing edge case if such functions started to error.


[SSI] Prepare mock files to return in case of error

Hi @noteboom,
as you already implemented in your code, in case of error we call:
https://github.com/CMCC-Foundation/bluecloud-mockup_method/blob/4ef7f95ce73e28fdf95dfd5c9646393ed3c6b5dc/log/logmng.py#L76
This function, is used to finalize the method in case of error.
In case of error, the most important thing to do is to provide to WPS the expected output files declared in the method interface (in software importer). If the outputs are present, also in case of error the web interface will be able to download the log data in order to have an information of the status of the execution ( or the message error).
To do this, it is sufficient to prepare some empty files with the same name of the expected output and then copy that file in the root directory in case of error. The handle_exec function already do this but with the OceanClimate outputs, as you can see here:
https://github.com/CMCC-Foundation/bluecloud-mockup_method/blob/4ef7f95ce73e28fdf95dfd5c9646393ed3c6b5dc/log/logmng.py#L91-L92
For your method, I suggest to create the mock directory and put here two output files with the same name of the expected ones (they can be also empty) and to update these two lines of codes in order to provide them in case of error

Hi @anto6715
Thanks for your message. I will update the SSI method accordingly.
Best regards,
JW

Hi @anto6715
The method now SSI_output.nc and SSI_area_timeseries.png (and output.json) as the output files... Should I changed this to output.nc and output.png (and output.json) in the SSI software and  method parameters?
JW

Hi @anto6715
The method now SSI_output.nc and SSI_area_timeseries.png (and output.json) as the output files... Should I changed this to output.nc and output.png (and output.json) in the SSI software and method parameters?
JW

No, the name must remains the same of your method. Here the code in your case:
shutil.copy("./mock/SSI_output.nc", "SSI_output.nc")
shutil.copy("./mock/SSI_area_timeseries.png", "SSI_area_timeseries.png")


About group items

Hi, in grammar, what's the rule for "word and some special characters"
Example
I'm sad.

The goal is to have "I'm" and "sad".
Using %import common.WORD gives UnexpectedCharacters error
I tried to use group item but not sure about the syntex.
wordset: ("I" "m" "'")

UnexpectedCharacters: No terminal defined for 'I' at line 1 col 1

That is because ' isn't considered part of a word. Try something like:
ENG_WORD: WORD ["'" WORD]

Closing the issue, but feel free to continue the discussion or ask more questions

Thanks for helping. I do have one question about number combined with words.
First everything works well with grammar:
ENG_WORD: WORD ["'" WORD]
Then I encounter strings like "100 students". The goal is to parse it into two ENG_WORD: 100, students.
Also how about word with number in it, like "ww2" and "Iphone8"
I have tried ENG_WORD: WORD ["'" WORD] [NUMBER WORD], it won't work.
(I'm so noob!)

What exactly is the result you're hoping for?
Do you want ww2 to parse into WORD, or WORD NUMBER?

my goal is that any consecutive combination of letter and number should be parsed into a "word".
special examples like
I'm happy should be parsed into 2 words I'm and happy.
book about ww2 should be parsed into 3 words book, about and ww2

Sounds like you just need to define word yourself:
WORD: (LETTER | DIGIT) +

%import common (LETTER, DIGIT)


Can't sign in with Google

Email sign in works, but when I try to sign in with Google, I'm getting a toast "Code: 10, Message: 10:" and nothing happens.

Google Sign in from the web app (using firebaseui-react) works, so the Firebase/Google setup should be correct.

Google Sign in from the web app (using firebaseui-react) works, so the Firebase/Google setup should be correct.

Hi @fabyeah, the issue is with android or iOS?

Hi @fabyeah, the issue is with android or iOS?

@oijusti I tested it with android device

@oijusti I tested it with android device

It works on iOS.
To make it work on android, it seems like one has to add "SHA certificate fingerprints" to the Android App in the Firebase console project settings.

https://stackoverflow.com/questions/51314476/firebase-auth-ui-google-sign-in-code10-message10
https://stackoverflow.com/questions/51360250/firebase-ui-authentication-with-google-fails-with-message-code10-message10/51360406
https://stackoverflow.com/questions/60049668/how-to-solve-code-10-message-10-with-google-signing-option

Kind of cumbersome if you just want to play around and test an app locally without uploading it to Google Play... 🙁

It works on iOS.
To make it work on android, it seems like one has to add "SHA certificate fingerprints" to the Android App in the Firebase console project settings.

https://stackoverflow.com/questions/51314476/firebase-auth-ui-google-sign-in-code10-message10
https://stackoverflow.com/questions/51360250/firebase-ui-authentication-with-google-fails-with-message-code10-message10/51360406
https://stackoverflow.com/questions/60049668/how-to-solve-code-10-message-10-with-google-signing-option

Kind of cumbersome if you just want to play around and test an app locally without uploading it to Google Play... 🙁

Good catch!

Good catch!

Just for posterity: I actually had the wrong SHA1 set in Firebase. So if you encounter this error, make sure you get the SHA1 from the debug.keystore in your project's android/app folder, NOT from some rogue debug.keystore for example in /Users/<username>/.android/debug.keystore. You can use keytool -keystore debug.keystore -list -v in the project's android/app folder to get the SHA1.
So for development you probably only need that one SHA1 (haven't tried it).


Update artist invite and editorial header images

          1 Warning

:warning:
Please provide a linked tracker story for this PR in the description

  Generated by :no_entry_sign: Danger