About last_index MPC.py

Hi, Hongzi
Thank you for your great work.
I have a question about implementation in test/mpc.py
Shouldn't last_index in line 177 be the index of last chunk?
last_index = int(CHUNK_TIL_VIDEO_END_CAP - video_chunk_remain)
If so, shouldn't last_index should start with 0?
last_index is used while calculating the max QoE using the bandwidth prediction. But it seems chunk sizes used in combo are one index moved back(last_index = current downloaded video index). So I think it should start with 0 until 47.But in the code it starts from 1.
This gives bitrate difference in the middle of video streaming a bit and usually matters with the last chunk, which the original code in github tries a higher bitrate at end, but if I modify the code it doesn't try a higher bitrate and results in lower rebuffer. Actually tring a higher bitrate doesn’t give benefit because of smoothness. Average reward varies by 2.6 (40.06 to 42.67 in simulation)
Line 177-180 to below
last_index = int(CHUNK_TIL_VIDEO_END_CAP - video_chunk_remain)-1
future_chunk_length = MPC_FUTURE_CHUNK_COUNT
if (CHUNK_TIL_VIDEO_END_CAP-1 - last_index < 5):
            future_chunk_length = CHUNK_TIL_VIDEO_END_CAP - last_index-1

Also, in rl_server/mpc_server.py

In line 191-192
if ( TOTAL_VIDEO_CHUNKS - last_index < 5 ):
                    future_chunk_length = TOTAL_VIDEO_CHUNKS - last_index

shoudln't it be
if ( TOTAL_VIDEO_CHUNKS - last_index-1 < 5 ):
                    future_chunk_length = TOTAL_VIDEO_CHUNKS - last_index-1

tested in realworld, 142 dropbox test_set, and found 1.3 QoE average increasing when fixed, compared original robust_mpc_server.py code and fixed code.

Thank you!

Thank you for your discovery! You might be right, this can be a potential improvement. We did the reimplementation of MPC the best we could but might miss some corner cases. This is for simulation though, the real evaluation code is in javascript. In a very high level, I would say this also showcases human engineering heuristic can sometimes be tedious and prune to error (like we did) while a machine based approach is more automatic and alleviate much of this burden.

Understood, Thank you!


Do not try to proxy edited messages if they start with the prefix

just realized this is rendered obsolete by #97


Add Reflux Architecture option

@yonatanmn thanks for the pull request. The tests are failing though, could you ensure they pass first?

Thanks @yonatanmn


ref(catalog): replace refs to mock mesh catalog with mock computer interfact

Description: Replaces calls to NewMockMeshCataloger with NewMockInterface to phase out the mesh cataloger interface. Part of #5156
[WIP] - remove call from from rds_test.go

Testing done: Changes to test suite -- ensured existing test scenarios still pass

Affected area:

Functional Area

Tests
[x]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project? no

Did you notify the maintainers and provide attribution? no

Is this a breaking change? no

Has documentation corresponding to this change been updated in the osm-docs repo (if applicable)? n/a

Closing in favor of #5233


Parsing error encountered

:tmi.twitch.tv 001 fcksentinal :Welcome, GLHF!
:tmi.twitch.tv 002 fcksentinal :Your host is tmi.twitch.tv
:tmi.twitch.tv 003 fcksentinal :This server is rather new
:tmi.twitch.tv 004 fcksentinal :-
:tmi.twitch.tv 375 fcksentinal :-
:tmi.twitch.tv 372 fcksentinal :You are in a maze of twisty passages, all alike.
:tmi.twitch.tv 376 fcksentinal :>
@badge-info=;badges=;color=;display-name=fcksentinal;emote-sets=0,300374282;user-id=530027885;user-type= :tmi.twitch.tv GLOBALUSERSTATE

We will not help you if you ignore the issue template,
and nothing of what you posted hints at an issue anyways.
Open a new issue or reopen this one with proof of a problem and the steps to reproduce it


C# integration.

This is an awesome project
Would like to know if there is an implementation on how to use C# code and send a message to the browser then vise versa.
Is there a working sample that I can see ?
Thank you!

Hi @stephenjason89. Nothing that I know of. Maybe search for "electron c# integration". Good luck!


Add my email to CommComm team meetings

Hi,
Can someone with Foundation Calendar Permission add my email<EMAIL_ADDRESS>to CommComm meetings?
I see there are already some people invited.
Maybe we should invite all the members of CommComm with their email. And make this step part in the onboarding process for new members. This way, when you share your calendar with someone, they can see that you are busy. It helps a lot.
I know there is a way to copy the events to own calendar, but that gets outdated when original event is updated.
Maybe someone can suggest a better solution.

@brianwarner can help with this.

Unfortunately this is one thing I can't help with... I believe the CommComm calendar invitation is maintained somewhere else.  I don't have it in the OpenJS Foundation general calendar.

I think @bnb can add.

Theoretically added. Let me know if it didn't work.

I think it didn't work as I didn't receive an invitation email. Also, I don't see it in the event.

Saved the event and it kept not adding you. I assumed it was sending you an email. Let me look into it.

Should have worked now.

Please, can I get added too?

It worked. 👍  thanks.


Update scalaz dependency version to 7.2.x series

Update the scalaz dependency to 7.2.4 and fix API calls where
scalaz has made breaking changes between the 7.1.x and 7.2.x series.

Thanks!


Update all dependencies

This is supposed to address deprecation warnings about using Node 12 and the set-output command. I also updated all deps in the test Java code and all actions in the workflow.
I had to update the fixture to change some names of Java packages after running tests on JDK 17 instead of JDK 8. I did remember to rebuild dist/index.js.

The fail is expected, as the action needs to test reporting on failures.
We could test it in SEP, but we have tests here that prove the action executes successfully. We can always pin it in SEP to a more specific version if anything breaks.


Add pattern escape for glob matching cargo clean files

Closes #10068.
Runs glob::Pattern::escape on path input to rm_rf_glob(). This fixes a bug in which cargo clean -p fails to delete a number of files from target/ if the path to target contains glob characters.

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @Eh2406 (or someone else) soon.
Please see the contribution instructions for more information.

Ah, you need to make sure to escape only the part of the path that isn't intended to be a pattern. The code that calls rm_rf_glob specifically expects one part of the pattern to actually be a glob (something like {pkg.name()}-*), and we'll need to keep supporting that (which is probably what the failing test is indicating).

Ah, you need to make sure to escape only the part of the path that isn't intended to be a pattern. The code that calls rm_rf_glob specifically expects one part of the pattern to actually be a glob (something like {pkg.name()}-*), and we'll need to keep supporting that (which is probably what the failing test is indicating).

Sorry about that! I was under the impression that only I could see draft PR's... I just pushed the correct solution.
Here's the output of running your script on my machine:
➜  cargo_test rm -rf glob-in-rm && ./reproduce.sh
~/Projects/cargo_test/glob-in-rm/hello ~/Projects/cargo_test/glob-in-rm
     Created binary (application) `foo` package
    Checking foo v0.1.0 (/Users/russweas/Projects/cargo_test/glob-in-rm/hello/foo)
    Finished dev [unoptimized + debuginfo] target(s) in 0.65s
~/Projects/cargo_test/glob-in-rm
~/Projects/cargo_test/glob-in-rm/[hello] ~/Projects/cargo_test/glob-in-rm
     Created binary (application) `foo` package
    Checking foo v0.1.0 (/Users/russweas/Projects/cargo_test/glob-in-rm/[hello]/foo)
    Finished dev [unoptimized + debuginfo] target(s) in 0.33s
~/Projects/cargo_test/glob-in-rm
Files [hello]/foo/target/.rustc_info.json and hello/foo/target/.rustc_info.json differ

Nice! Draft PRs are visible to everyone, they are just marked as "not ready to merge". The idea of a draft PR is to get feedback on work that's not yet completed, or just to have a place to allow discussion on a fix before it's done. I think it was totally fine to open this as a draft :)

Nice! Draft PRs are visible to everyone, they are just marked as "not ready to merge". The idea of a draft PR is to get feedback on work that's not yet completed, or just to have a place to allow discussion on a fix before it's done. I think it was totally fine to open this as a draft :)

Now I know 😆. At my internship I used draft PR's like a Todo list of things I was working on, so I would make a tiny change then push it.

Note: would also need to incorporate these changes into #10070
https://github.com/rust-lang/cargo/blob/2f375dab7c563de95ee2d835b0304ae8e307a600/src/cargo/ops/cargo_clean.rs#L176-L184

Tests passed, pushed final commit to undo superfluous formatting changes

Thanks for this! Would you be up for adding some tests for this as well?

And just squashed the commits, I noticed it was getting excessive

Looks like the test works on windows-latest, nightly-gnu, i686-pc-windows-gnu , but not windows-latest, stable-msvc, i686-pc-windows-msvc. Specifically, the build artifacts look like they're being generated differently...

@bors: r+
Thanks!

:pushpin: Commit effc72042b7f4edb40105750e7b9723e9a854d72 has been approved by alexcrichton

:hourglass: Testing commit effc72042b7f4edb40105750e7b9723e9a854d72 with merge adf601d41c6a72da3925c339300fb27f41da9baf...

:sunny: Test successful - checks-actions
Approved by: alexcrichton
Pushing adf601d41c6a72da3925c339300fb27f41da9baf to master...


In practice, Azure AD Device Limits seem to be 'per device type'

It seems as though the maximum number is per device type (i.e. Windows, Android, etc.).  It's either this, or "Azure AD Registered" is also immune, not just "Hybrid Azure AD Joined".  I have confirmed in multiple tenants, if you have the limit set to 1, you can have both an Azure AD Registered Windows 10 device (NOT Hybrid Azure AD Joined), and then you can still add another device of a different type.
Is this something that can be documented, or is this an issue with the product?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a8119688-e88d-2916-3dcf-bf777a5fc80c
Version Independent ID: e767d695-1c9f-64e4-e6c4-fef07b0f3d2e
Content: How to manage devices using the Azure portal
Content Source: articles/active-directory/devices/device-management-azure-portal.md
Service: active-directory
Sub-service: devices
GitHub Login: @MicrosoftGuyJFlo
Microsoft Alias: joflore

@JeremyTBradshaw We are checking this and we will get back to you.

No maximum number of devices per user includes all OS versions. If its set to '1', a user will not be able to register more than a single device . I just tested this in my lab and its working as expected.
I can look into this further if needed. What is your configuration, and how are you trying to register your device?
Best,
James

Hi @James-Hamil-MSFT , thanks for testing it out.  It's a customer I'm working with where I noticed this.  I believe I have it figured out.  They have most of their domain-joined computers registered in Azure AD, although they aren't Hybrid Azure AD Joined, and only show up in Azure AD as "Azure AD registered".  Most users' domain-joined computer show up this way.
In Azure AD Device Settings: Maximum number of devices per user, the setting is 1.  However, users are able to register their Android device successfully, and they all then show up with 2 devices - their Windows/domain-joined computer, and their Android device.
My assumption is that the domain-joined are being treated as Hybrid Azure AD joined, which are not counted towards the maximum setting.  I think it's minor details here, I could probably have just skipped opening the issue.  I'll leave it open for you to close in case you'd like to confirm this behavior.
Thanks,
Jeremy

Hi @JeremyTBradshaw , I'm glad you figured it out! It looks on my end to be confirmed as well. Please reach out if you need any more assistance!
Best,
James


Provide a Root CA when creating/updating service broker

When I register or update a service broker, I'd like to offer https:// URLs to my broker but the SSL certs are often self-signed. I'd like to be able to have users register my broker with the matching root CA for the SSL/TLS certs please.
Currently, if the service broker has self-signed certs then I'm required to downgrade to accessing/registering the broker via http://

@drnic We're putting together a track of work to remove the skip_cert_verify option such that we are secure everywhere by default.  The service broker interaction is one of the stickier ones since we can't supply CAs in a bosh manifest.  This sounds like it could be our solution.

Hey @drnic
We're just taking a look at this issue; is this still something that is important for you?
Thanks!

It’s only important to me if it’s important to you :)

From: Matt McNeeney<EMAIL_ADDRESS>Sent: Monday, September 3, 2018 8:16 pm
To: cloudfoundry/cloud_controller_ng
Cc: Dr Nic Williams; Mention
Subject: Re: [cloudfoundry/cloud_controller_ng] Provide a Root CA when creating/updating service broker (#825)
Hey @drnichttps://github.com/drnic
We're just taking a look at this issue; is this still something that is important for you?
Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/cloudfoundry/cloud_controller_ng/issues/825#issuecomment-418069563, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAAAbGOV0z4I0h3R-1rzfBKBy4MBObslks5uXQE4gaJpZM4NjRiW.

This still sounds valuable to me.  We've been working on turning http to https everywhere we can.  This can be done for brokers if the brokers are using a cert that the vm cc is running on trusts it, but doesn't allow operators to easily opt-in to self-signed certs without putting them into the cc vm trust store.

I think that platform to service broker communication still supports both
HTTP and HTTPS though. Are/were there plans to disable HTTP?
On Wed, 12 Sep 2018 at 22:35, Zach Robinson<EMAIL_ADDRESS>wrote:

This still sounds valuable to me. We've been working on turning http to
https everywhere we can. This can be done for brokers if the brokers are
using a cert that the vm cc is running on trusts it, but doesn't allow
operators to easily opt-in to self-signed certs without putting them into
the cc vm trust store.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/cloudfoundry/cloud_controller_ng/issues/825#issuecomment-420805919,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AG7UzC6ecVz1WaXhnE9k551Mdgr2LnT6ks5uaX4cgaJpZM4NjRiW
.

Closing for now, but will reopen if this becomes a blocker for anyone!

Reopening this since cf-for-k8s redirects http to https by default. This makes this and issue since users can't just downgrade to http when they're trying to interact with on-platform brokers.
See this Slack convo where @reneighbor ran into this today:
https://cloudfoundry.slack.com/archives/C017LDM6KTQ/p1602280933144500
cc @piyalibanerjee @Gerg

Now that we (the cf-for-k8s team) understand this better, we're thinking that we made need a K8s implementation of this: https://bosh.io/docs/trusted-certs/, so that we could potentially use a self-signed CA by adding it to the trust store for [at least] the necessary containers.
Does that make sense to you, cake folks?

Closing this since cf-for-k8s is no longer under active development


Please add farming timers

Farming timers to know when your herbs are done are essential. Please add.

There already is a pull request (meaning that someone already wrote the code for the plugin, but it might needs some additional work) for a farming plugin (#1244), so it should be coming sometime soon.

Please search for duplicate issues before posting a new one :)
#511


Single table inheritance

See https://github.com/jgrossi/corcel/pull/73 for details and discussion of this pull request

Please, paste the phpunit result here just to store that everything is ok with the tests :-)

Great job with this! :) I will definitely use this for ACF and so many things..
:+1: :+1:

PHPUnit 4.2.2 by Sebastian Bergmann.

Configuration read from C:\xampp\htdocs\corcel\phpunit.xml

Time: 188 ms, Memory: 8.00Mb

OK (31 tests, 210 assertions)

Process finished with exit code 0```

Very nice job @mattKendon thanks again!

Happy to help, @jgrossi. I Found I needed this functionality and thought that others might as well.

Very nice! :-)


fix(interactive): Fix test_gremlin_timeout in Python Unit Tests

What do these changes do?
as titled.

Related issue number

Fixes #3480

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (ebcf431) 24.33% compared to head (4a4ca29) 45.37%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3536       +/-   ##
===========================================
+ Coverage   24.33%   45.37%   +21.03%
===========================================
  Files         176      176
  Lines       16212    16167       -45
===========================================
+ Hits         3945     7335     +3390
+ Misses      12267     8832     -3435

Files
Coverage Δ

python/graphscope/tests/unittest/test_lazy.py
96.45% <ø> (+96.45%)
:arrow_up:

... and 70 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ebcf431...4a4ca29. Read the comment docs.


Ability to write length/size of streams to log files

Reporter
Laurens/Richard
Is your feature request related to a problem? Please describe.
For the Ibis4Titan project Richard would like to log the size or length of a stream to a log file, for performance anlyses.
Describe the solution you'd like
That you could use the SizePipe to retrieve the actual size of a stream or another solution.
Describe alternatives you've considered
This is what Richard tried:
<SizePipe
name="getFileSize"
getInputFromSessionKey="ContentStream"
storeResultInSessionKey="FileSize"
preserveInput="true"

Additional context
Besides this issue, it seems to be related to not seeing any size statistics anymore:

The problem is that when you want to determine the size of a steam you will have to store/read the stream, as a non-stream, and then convert it back to a stream. Usually when you use streams, you don't want to store it.


Clerk cannot render wrapped values

;; # 🐞Debug
(ns scratch.debug
  {:nextjournal.clerk/visibility {:code :hide}}
  (:require [nextjournal.clerk :as clerk]
            [nextjournal.clerk.viewer :as viewer]))

{:nextjournal/value (clerk/with-viewer viewer/string-viewer "hello")}

the above gives:

This is the cause of https://github.com/nextjournal/clerk/issues/638#issuecomment-2018168649.


Update org.eclipse.jdt.core to 3.32.0

Updates org.eclipse.jdt:org.eclipse.jdt.core from 3.24.0 to 3.32.0.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.eclipse.jdt", artifactId = "org.eclipse.jdt.core" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "@monthly" },
  dependency = { groupId = "org.eclipse.jdt", artifactId = "org.eclipse.jdt.core" }
}]

labels: library-update, early-semver-minor, semver-spec-minor, commit-count:1

Superseded by #1711.


1278 revisit

https://github.com/rancher/dashboard/issues/1278

H3 and H4 are both used about evenly (and probably inconsistently); changing all ArrayLists to H4 is going to make something else not match somewhere...


Feature 2.0.0/rkst 1 api updates for debugging

closing as there are additional updates to deal with.


Find stopped containers when restart param is given

this should also fix gh-305

@paschdan thanks, much appreciated.


Fix for the issue 18806: InputRange for std.algorithm.minIndex

Fixes https://issues.dlang.org/show_bug.cgi?id=18806
I'll add a changelog when this is ready to be merged.

Please change the commit message to include "fix issue 18806 - (rest of description)" so that the dlang bot picks up on it. I'm not sure what caused the linux failure, looks random?

A changelog will be automatically added because this is a closed issue. Changlelogs are only necessary when users need to know more, such as how to work around issues with the fix.

What about maxIndex?
Also, std.internal.test.dummyrange helps a lot with avoiding the annoying work of defining test ranges.


Fix dimension order in the image shape

The order of the dimensions did not matter that much in the 0.5.* version, since it's rarely used. But in 0.6 we use the output shape to create tensors passed to the models, so it's important to have this correct. In quite a few models input width is equal to the input height, but for the Ultraface models these dimensions are different, so the FaceDetectionWithVisualization.kt example will fail in master branch without the fix.

I checked once again, everything seems correct to me.


Fixed dispatch queue usage if OS_OBJECT_USE_OBJC is deactivated.

I have legacy code that must run on iOS 4.3 but dispatch objects are not bridged with Objective-C on this system. Thus, i need to define OS_OBJECT_USE_OBJC to zero to deactivate bridging between dispatch objects and obj-c.
So this patch fix serial queue stuff when bridging is deactivated. I also fixed properties synthesis when compiling with -Wobjc-missing-property-synthesis option (I love to compile with -Wall -Wextra options 😝).
Finally, I added API for custom management of the serial queue and the reachability changed event.
Regards.

ping


Feat: block users from accessing "Stop" button in the observation request form if they are not the current requester

Current behavior
As of right now, the application does not block other users from accessing the observation request form. If the current user requests an observation and it proceeds, other users connected to the same system can still access the request form, I think.
Expected behavior

When no observation is taking place, all users can access the form.
When an observation is taking place
a. the current observer can use the "Stop" button
b. all other users cannot access the "Start" or "Stop" buttons

Related: #103

Initialized by 4cf1098e99fdb25a1eeb20f3744c80dab05275f8


NodeJS+CharkaCore, ERROR in 1.47.0

I'm using node-chakracore for some reason.
but Now, I cannot debug node-chakracore with VSCode.

VSCode Version: 1.47.1
OS Version: Windows 10

Steps to Reproduce:

Make launch.json with VSCode
Download node-chakracore release
Set "runtimeExecutable" to node-chakracore
Launch it!

Does this issue occur when all extensions are disabled?: "Yes"
Problem:
\resources\app\extensions\ms-vscode.js-debug\src\bootloader.bundle.js contains spread in object literals and chakracore cannot parse it.
node-chakracore splits "NODE_OPTIONS"(envvar) with whitespace and VSCode put a file path with whitespace. (the problem of node-chakracore)

Thanks for the issue. We don't make any effort to support node-chakracore, which had been archived by the Node team and is no longer maintained. I recommend either:

Using Node proper
Setting debug.javascript.usePreview: false to use the old debugger. This option will eventually go away, but may give you time to update.

Thanks for your help :)


Fixed duplicate label translations for secondary itemsets and use itextID for selects with choices that have media specified

Closes #464, #463
Why is this the best possible solution? Were any other approaches considered?
What are the regression risks?
None.
Does this change require updates to documentation? If so, please file an issue here and include the link below.
No
Before submitting this PR, please make sure you have:

[x] included test cases for core behavior and edge cases in tests_v1
[x] run nosetests and verified all tests pass
[x] run black pyxform to format code
[x] verified that any code or assets from external sources are properly credited in comments

Codecov Report

Merging #467 into master will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #467      +/-   ##
==========================================
+ Coverage   83.39%   83.40%   +0.01%
==========================================
  Files          25       25
  Lines        3601     3604       +3
  Branches      836      837       +1
==========================================
+ Hits         3003     3006       +3
  Misses        454      454
  Partials      144      144

Impacted Files
Coverage Δ

pyxform/question.py
92.95% <100.00%> (+0.10%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 50921a8...627cc08. Read the comment docs.

Thanks! It would be really helpful to get a high-level view of the decisions you made and some risk analysis. There are clearly ramifications if downstream tools use the itext ids in some way. I don't know how likely that is. Did you think through that?
If we're going to be changing the structure of the itext ids, we should consider alternatives. For example, we could use uuids for those identifiers instead of using the list and choice names. I don't really understand the static_instance prefix. If it's just noise, could we remove it?

I didn't think this would have any ramifications to be honest which might have been a rather naive viewpoint on my part. It seemed like some itext ids for choices where being generated in the static_instance-<list_name>-<index> format when a choice had multiple languages or choice filters and others generated using translation path of the element which didn't seem quite consistent which led to additional itext entries being created in other code locations. The static_instance format seemed a bit more easier to predict as such I kind of leaned towards that... I believe we can remove it... I'll work on this.

Closing this in favor of #468. I didn't probably document my steps here or my reasoning for some of the changes. Most changes made here are actually not needed in order to fix the tagged issues. #468 is cleaner and I believe a bit more documented.


move the topic box under the message box

usually messages are sent to a default topic (no topic) so it would be more convenient to put the topic box under the message box,
by mistake I write the message in the topic box many times.

Thanks for the feedback.
There are definitely things we can do to better avoid this sort of mixup. I think we will do some of those in our upcoming new app:

https://github.com/zulip/zulip-flutter/issues/493
https://github.com/zulip/zulip-flutter/issues/494

I don't think the specific solution of swapping the layout in this way will be something we do, though. It's the inverse of what Zulip web does and of how the messages appear after they're sent (where the topic always goes above the content), and I'm not particularly convinced it would make this issue better instead of worse.


Bad URL on wekeo provider

Describe the bug
An error occurs when requesting wekeo provider
Code To Reproduce
auth = dag._plugins_manager.get_auth_plugin("wekeo").authenticate()

Output
AuthenticationError: Could no get authentication token: HTTPSConnectionPool(host='wekeo-broker.apps.mercator.dpi.wekeo.eu', port=443): Max retries exceeded with url: /databroker/gettoken (Caused by ResponseError('too many 502 error responses')),

Environment:

Python 3.11.0
eodag (Earth Observation Data Access Gateway): version 2.11.0

Bugfix
The wekeo url has changed : changing url from 'https://wekeo-broker.apps.mercator.dpi.wekeo.eu/databroker/' to 'https://wekeo-broker.prod.wekeo2.eu/databroker' in the provider configuration fixes the bug.

hello @ansotoo and thanks for submitting this issue

Bugfix
The wekeo url has changed : changing url from 'https://wekeo-broker.apps.mercator.dpi.wekeo.eu/databroker/' to 'https://wekeo-broker.prod.wekeo2.eu/databroker' in the provider configuration fixes the bug.

this url did not last long. Instead, wekeo moved to a new API, see https://github.com/CS-SI/eodag/issues/999


Couldn't access weights file after training

Hi,
I tried to train the model on QEMU-v8 with OP-TEE 3.10.0. After the process finished, it tried to save the weights, but couldn't open the file in the tmp/backup directory. Here's the log:
# darknetp classifier train -pp_start 2 -pp_end 10 cfg/mnist.dataset cfg/mnist_l
enet.cfg
Prepare session with the TA
Begin darknet
mnist_lenet
1
layer     filters    size              input                output
    0 conv      6  5 x 5 / 1    28 x  28 x   3   ->    28 x  28 x   6  0.001 BFLOPs
    1 max          2 x 2 / 2    28 x  28 x   6   ->    14 x  14 x   6
    2 conv_TA    6  5 x 5 / 1    14 x  14 x   6   ->    14 x  14 x   6  0.000 BFLOPs
    3 max_TA       2 x 2 / 2    14 x  14 x   6   ->     7 x   7 x   6
    4 connected_TA                          294  ->   120
    5 dropout_TA    p = 0.80                120  ->   120
    6 connected_TA                          120  ->    84
    7 dropout_TA    p = 0.80                 84  ->    84
    8 connected_TA                           84  ->    10
    9 softmax_TA                                       10
   10 cost_TA                                          10
workspace_size=235200
Learning Rate: 0.3, Momentum: 0.9, Decay: 5e-05
3000
28 28
output file: /media/results/train_mnist_lenet_pps2_ppe10.txt
current_batch=0
Loaded: 0.219263 seconds
1, 0.033: 0.000000, 0.000000 avg, 0.298802 rate, 5.330773 seconds, 100 images
user CPU start: 1.535481; end: 7.408529
kernel CPU start: 3.927328; end: 3.927328
Max: 14976  kilobytes
vmsize:1037248; vmrss:13416; vmdata:53548; vmstk:132; vmexe:432; vmlib:2280
Loaded: 0.000431 seconds
2, 0.067: 0.000000, 0.000000 avg, 0.297607 rate, 5.216192 seconds, 200 images
user CPU start: 7.413332; end: 12.828843
kernel CPU start: 3.927328; end: 3.927328
Max: 19840  kilobytes
vmsize:1037248; vmrss:13776; vmdata:53812; vmstk:132; vmexe:432; vmlib:2280

At the end of the process:
Loaded: 0.000476 seconds
1000, 33.333: 0.000000, 0.000000 avg, 0.000000 rate, 4.992810 seconds, 100000 images
user CPU start: 4903.527241; end: 4908.466377
kernel CPU start: 331.553209; end: 331.853131
Max: 24952  kilobytes
vmsize:1037180; vmrss:16744; vmdata:55588; vmstk:132; vmexe:432; vmlib:2280
Saving weights to tmp/backup/mnist_lenet.weights
Couldn't open file: tmp/backup/mnist_lenet.weights

Then I realized there was no tmp/backup directory in root. So, I created tmp inside root and backup inside tmp. Then I ran the program again and this time I got no errors. But I couldn't find the weights file anywhere.
Loaded: 0.000713 seconds
1000, 33.333: 0.000000, 0.000000 avg, 0.000000 rate, 5.079499 seconds, 100000 images
user CPU start: 4939.497528; end: 4944.462404
kernel CPU start: 350.172044; end: 350.464667
Max: 24452  kilobytes
vmsize:1037228; vmrss:16260; vmdata:55940; vmstk:132; vmexe:432; vmlib:2280
Saving weights to tmp/backup/mnist_lenet.weights

From the QEMU terminal, I could see there's a mnist_lenet.weights file hidden in the tmp/backup with command ls -a.
But is there anyway to access the weights file? I tried to copy it to other directories outside the root but failed.
Any help would be really appreciated. Thanks.

Hi @shibz-islam
I have the same problem.
So have you solved this problem?

Hi @James567
I was able to access the weights file by sharing folder between host and guest OSes.
You will find the necessary info here:
https://optee.readthedocs.io/en/latest/building/devices/qemu.html?highlight=VirtFS%20QEMU#host-guest-folder-sharing
I hope this is the right approach!

@shibz-islam thank you very much , hope this is the right approach too!

Hi @James567 I was able to access the weights file by sharing folder between host and guest OSes. You will find the necessary info here: https://optee.readthedocs.io/en/latest/building/devices/qemu.html?highlight=VirtFS QEMU#host-guest-folder-sharing I hope this is the right approach!

Hello！I follow your instructions, but I still can't see it in the folder, but use ls can see it.  Could you please help me solve it?

HI @Xxxgrey
make QEMU_VIRTFS_AUTOMOUNT=y run

I use this command and it works for me.

HI @Xxxgrey
make QEMU_VIRTFS_AUTOMOUNT=y run

I use this command and it works for me.

Thanks a lot .I'm going to try it.


Fix invalid Xml comment being generated in preprocessor if "--" is in the path to a resolved Sdk

Looks like there was one place in the preprocessor where it wasn't replacing "--" with "--" in Xml comments.
@rainersigwald

@dotnet-bot test Windows_NT Build for Desktop please


Meet/hotfix

Reviewers Checklist

[ ] Appropriate logging is done.
[ ] Appropriate error responses.
[ ] Handle every possible exception.
[ ] Make sure there is a docstring in the command functions. Hint: Lookout for
botcmd and re_botcmd decorators.
[ ] See that 100% coverage is there.
[ ] See to it that mocking is not done where it is not necessary.

ack 02e576870b71a7581093255ce1bd0a04ad2e60a0  ab93c8897cc5d993aee2980d287ca0e7bfd33531

@rultor merge

@rultor merge

@meetmangukiya OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge

@meetmangukiya Done! FYI, the full log is here (took me 2min)


Fix link to source

Hi.
Sometimes files are incorrect. Especially the link to alert me hurt.
Yours faithfully

Thanks! :)


Rethink ranking computation

Right now, the computation of the rankings happens on the website in javascript over here:
https://github.com/FastEval/FastEval/blob/02ca1e9d60804bc05def65696815dec628fbbbcc/website/benchmarks/main.js#L27
I would prefer to do this in python instead. The current algorithm is also influenced by the fact that I wrote it in javascript and didn't have some python packages available that would have probably changed my choice on what method to use and also simplified the code.
However, the problem with doing it in python is that it would mean there would be some file in the end that would contain those rankings so that we could fetch that file in javascript to display the result on the website. This file that contains the rankings would be very susceptible to merge conflicts if there are multiple branches with different additional models evaluated, which would e.g. be the case if people would evaluate models independently from some version and then submit pull requests with that. Resolving these merge conflicts would require recomputing the rankings before merging the results.
It would be more complex than simply being able to merge without anything like that. So that's an argument against doing it in python. But on the other hand, javascript is really not the right language for writing this kind of code. And with many models, this computation can actually take a bit of time if many models have only been evaluated on a subset of the benchmarks, so it's not something that should happen on every page load.

I'm going to leave it in the website for now. Probably move the code to a separate file and clean it up and also make it use some seeded randomness, but generally it's going to remain part of the website code due to the merge conflicts otherwise.


PdfViewerActivity Error

Using this in java and i am unable to get PdfViewerActivity, as it is showing not found error...(Dependency Added)

Hi Mukul, Thanks for checking out the repo.
I am sure if you have added the dependency currently, you will be able to import PdfViewerActivity.
But in case of Java, to call "buildIntent" method you need to use
PdfViewerActivity.companion.
Please check the dependency and implementation again and let me know.

Can you please send whole implementation in java? How to use this library
in java?
On Sun, 13 Dec, 2020, 11:02 pm Rajat<EMAIL_ADDRESS>wrote:

Hi Mukul, Thanks for checking out the repo.
I am sure if you have added the dependency currently, you will be able to
import PdfViewerActivity.
But in case of Java, to call "buildIntent" method you need to use
PdfViewerActivity.companion.
Please check the dependency and implementation again and let me know.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/afreakyelf/Pdf-Viewer/issues/5#issuecomment-744040954,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AP6H5XOLVM5UDTR6WTDXWDTSUT3CHANCNFSM4UZU5DSA
.

Hi Please try this way.
implementation 'com.github.afreakyelf:Pdf-Viewer:1.0.3'

launch_pdf.setOnClickListener(v -> {
        startActivity(
                PdfViewerActivity.Companion.buildIntent(
                        this,
                        "pdf_url",                                // PDF URL in String format
                        false,
                        "Pdf title/name ",                        // PDF Name/Title in String format
                        "pdf directory to save",                  // If nothing specific, Put "" it will save to Downloads
                        false                    // This param is true by defualt.
                )
        )
  });

Thanks brother, it worked... please one more request..how can i save my pdf
file in cache memory because i need to open that file again and again and
it takes lot of time and internet for opening.
On Mon, 14 Dec, 2020, 11:04 am Rajat<EMAIL_ADDRESS>wrote:

Hi Please try this way.
implementation 'com.github.afreakyelf:Pdf-Viewer:1.0.3'
launch_pdf.setOnClickListener(v -> {
startActivity(
PdfViewerActivity.Companion.buildIntent(
this,
"pdf_url",                                // PDF URL in String format
false,
"Pdf title/name ",                        // PDF Name/Title in String format
"pdf directory to save",                  // If nothing specific, Put "" it will save to Downloads
false                    // This param is true by defualt.
)
)
});
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/afreakyelf/Pdf-Viewer/issues/5#issuecomment-744179799,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AP6H5XNEYIJKCC2JRGMBJLTSUWPXRANCNFSM4UZU5DSA
.

Saving file into cache is still in pending. Feel free to modify if you can and raise PR or else I will add by evening.
Don't forget to ⭐ repo.
Thanks


AMBARI-23847. User information is missing when upgrade wizard is laun…

…ched by a custom user.
What changes were proposed in this pull request?
User information is missing when upgrade wizard is launched by a custom user.
How was this patch tested?
manually
Please review Ambari Contributing Guide before opening a pull request.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/2279/
Test PASSed.


Only partially working out of the box on my ubuntu laptop

Cloned the repo, disabled any services attaching to the ports 1880, 1883 and 3000, then:
Do this with conda activated

pip install docker-compose
docker-compose up -d
big install
docker-compose logs -f

MQTT works (with these commands in different windows):

mosquitto_sub -h localhost -v -t "#"
mosquitto_pub -h localhost -t "hi" -m "there"

Node-RED now appears to work:

http://localhost:1880
You need to use "mosquitto" as the broker, not localhost.

The influxDB node is missing, so I can't make database data.
Jupyter also didn't come live for a while on http://localhost:8888/lab with password mingstack.  Then does after a few minutes.  It starts out with no syntax highlighting and nothing will run.  Now it does.
Grafana comes in on http://localhost:3000  default user/pass=admin/admin and then you are forced to change it.
I was hoping to run the following to connect+download from the grafana database using the following python code:

import influxdb
client = influxdb.DataFrameClient("<IP_ADDRESS>", 8086, "root", "root", "balena-sense")
k = client.query("select * from fridgepower order by time desc limit 10")

Some needs

Where is the /data file connected to externally that the jupyter lab could connect/load/save from?
install influxdb into python
Inclusion of node-red serial
Inclusion of pyserial in jupyter
Inclusion of jupyter-micropython-kernel
Provision the system with an initial flow (eg doing random numbers) saving to an initial database with an initial dashboard.

This was due to Julian being on an older version of docker (docker 17.03), not much could really be done. It might be worth putting some prerequisite dependencies in the Readme, such as what docker version should be in use and a link to how to install Docker.

@goatchurchprime can you retest this with whatever Docker version @MatthewCroughan says you need?

@ajlennon we already went through this in telegram, I had him uninstall all old versions of Docker then run https://get.docker.com/.
Should mark this issue as resolved once we document this in the README.

There are a bunch of other issues up there about things that need to be installed by default (like the InfluxDB nodes)

Things are getting close for an amazing package that knocks people out.  I'm looking for an instant installation of the full stack pre-loaded with all the tools on the go.  There may be multiple installs of the MING stack on personal computers, that can push data around via mosquitto brokers, until it's understood that one should do it with one lab install (properly backed up) that everyone connects to.
Having been around to some researchers, we need things the serial-ports to work, because that's how lab-equipment is often operated.  Also, there are some systems that only have windows interfaces (through the same serial-COM ports) that dump their data into a file.  Node-red should be able to tail those files and adjust its control behavour.  The jupyter-influxDB integration makes it possible to pick up and analyse the data from the experiments when they are run.
Chris says that wsl2 means we should be able to get docker in linux on windows almost natively.

The notes were made after we'd sorted out the docker version:

Docker version 19.03.5, build 633a0ea838

Docker-compose only runs when I have conda activated, then:

docker-compose version 1.25.4, build unknown

Overcoming missing components (that should be loaded by default)
Node-red:
Installing node-red-contrib-influxdb using manage pallette
This should have a couple of example flows to kick it off, like a repeating inject node that passes the following javascript payload into the influxdb database:

msg.payload = [ { "xxx":Math.random()*2+1}, {"yyytagtype":"zzztag"} ];

I am using database=aaadatabase, and measurement=bbbmeasurement to make it clear that those names are user-defined and not mistaken for keywords.
Influxdb:
This should boot up with one default database in it.  The http://localhost:8086 link, which is supposed to be a dashboard for it, gets 404 page not found.  I don't have the command line login to work it.  See Jupyter section for how I added a database.
Jupyter notebook:

The influxdb connection

[install the module]
import subprocess
print(subprocess.check_output(["pip3",  "install", "influxdb"]).decode())
[now restart kernel]

import influxdb
client = influxdb.DataFrameClient("influxdb")
client.create_database("aaadatabase")

[wait for the nodered to add a bunch of random values]
c = influxdb.DataFrameClient("influxdb", database="aaadatabase")
x = c.query("select * from bbbmeasurement order by time")
z = x["bbbmeasurement"]
display(z)

from matplotlib import pyplot as plt
plt.plot(z.xxx)

The serial port connection:

import subprocess
print(subprocess.check_output(["pip3",  "install", "pyserial"]).decode())

import serial, serial.tools.list_ports
lp = list(serial.tools.list_ports.grep(""))
lp.sort(key=lambda X: (X.hwid == "n/a", X.device))  # n/a could be good evidence that the port is non-existent
ports = [x.device  for x in lp]
print(ports)

Micropython

If we can get serial connection, we might as well install the micropython kernel.  There might be some other kernels to add in, eg Julia, which is a very popular language among academics.
Other node-RED components
Install node-red-node-serialport (this won't work till we solve it for the Jupyter container).
Install node-red-dashboard (hooray!  this works).
We should have a default node-flow showing off the dashboard with a couple of dials on it because getting it going

Yeah agreed. There's a bunch of stuff that should be installed out of the box on node-red

@ajlennon I disagree, it should be an optional or build argument, it should be a feature that can be activated at build time, but why would we install things by default for the user rather than letting them import their own stuff?

We could also make a different project that uses this project as a submodule, MING being a layer that is pulled in that adds all of these extraneous things.

Additionally, Jupyterlab is based on a balena image, so I'm really not sure how that's going to work on a generic system, will look into getting this working.

@goatchurchprime What you're looking for is not to modify the base image of MING so that everyone in the world ends up with a bunch of lab software in their MING install, but to fork MING and use it as a base to make your own embedded package that can be installed for your own needs, with your own extra software, which is what this is all about. I'll help you out with this in person tonight and show you what I mean.

The instructions you have provided are more than enough to get on with this, so I'll help you out with it and translate those instructions into a dockerfile in a fork of MING and show you what I mean by all of this.

Influxdb:
This should boot up with one default database in it. The http://localhost:8086 link, which is supposed to be a dashboard for it, gets 404 page not found. I don't have the command line login to work it. See Jupyter section for how I added a database.

The default database name is ming_default which is specified in the docker-compose file:
https://github.com/DynamicDevices/ming/blob/4c77867e654a5bb35ee6d450e5d0c6d5ca3fa8b8/docker-compose.yml#L19

@ajlennon I disagree, it should be an optional or build argument, it should be a feature that can be activated at build time, but why would we install things by default for the user rather than letting them import their own stuff?

The value of Docker / MING is that users can get going immediately without faffing about or having to understand what to configure or turn on or which buttons to push.
I see no downside in activities that people are likely to want to undertake (e.g. using serial ports) having support installed and ready to go...

@goatchurchprime What you're looking for is not to modify the base image of MING so that everyone in the world ends up with a bunch of lab software in their MING install, but to fork MING and use it as a base to make your own embedded package that can be installed for your own needs, with your own extra software, which is what this is all about. I'll help you out with this in person tonight and show you what I mean.
The instructions you have provided are more than enough to get on with this. Thanks.

Not at all keen on forking MING. There's no downside to having all this installed and ready to go in the standard build

In Node-RED set the Influx node to: influxdb:8086, database: ming_default
In Grafana we need a standardized password.  And then to be provisioned with a link to the ming_default database.  Then it needs a panel view into the
Use as a function to seed the database with:
Math.sin(Math.PI2(((new Date()).getTime()%60000)/60000) + 1 + Math.random()*0.2;
May seed with two functions to show how values can interact in the same table.
Where are these /data directories?  Can they get shared with the outer file system?
Jupyter-lab has a Terminal window, so we can install anything we need there, though it doesn't do the control-Up history commands.
Also can't run apt-get install, because I want Git to see if I can get micropython.
Version of JupyterLab may be out of date.  My personal one says this on startup:

Build Recommended
JupyterLab build is suggested:
jupyter-threejs needs to be included in build
jupyterlab-datawidgets needs to be included in build

This could mean that the ipyvolume examples might work:
https://ipyvolume.readthedocs.io/en/latest/
I wonder what happens if I do this docker-compose up a second time:
It just says it's all up to date.
May need the cheatsheet for closing down and opening up and updating the system.
Also, I'll need scripts to help backup the different components, etc.  The kind of guidance and workflows.

other keep safe commands:
Clone https://github.com/DynamicDevices/ming
In the directory, do:

docker-compose up -d

This shows what's up and running:

docker ps

To list the images:

docker images

To log what's going on, do:

docker-compose logs -f

To kill all docker containers and images:

echo "Nuking docker containers, images, and volumes." && docker stop $(docker ps -aq) || true && docker rm $(docker ps -aq) || true && docker rmi -f $(docker images -aq) || true && docker volume rm $(docker volume ls -q) || true

Also can't run apt-get install, because I want Git to see if I can get micropython.

Why not? Have you tried apt-get update first?

@goatchurchprime the hostname to use for servers is the name of the container (as you say above) rather than localhost. This is correct behaviour and should probably be written down somewhere

When we get some of these issues sorted out, I'll start a habit of building it up from scratch and tearing it down on different computers to prove it works.
To make these tests easier, we need to provision it with some exemplar node-red flows and dashboards to make these tests quicker.  Just try it out like this way for a while with the whole kitchen sink.
Then if it doesn't feel right we can fork off into derivations and all that complexity.  But not while the dividing line is unclear (eg should the ming_default database provision be in the clean upstream copy, or the enabled downstream copy?)  It just causes hassle of managing two repos and potentially fixing up both of them when stuff is in flux.

@goatchurchprime What you're looking for is not to modify the base image of MING so that everyone in the world ends up with a bunch of lab software in their MING install, but to fork MING and use it as a base to make your own embedded package that can be installed for your own needs, with your own extra software, which is what this is all about. I'll help you out with this in person tonight and show you what I mean.
The instructions you have provided are more than enough to get on with this. Thanks.

Not at all keen on forking MING. There's no downside to having all this installed and ready to go in the standard build

I am thinking now that the addition of obvious things like the influxdb node in Node-Red since we provide an InfluxDB container so it should be there by default.
But there are obvious things that should not be included by default because they have not much to do with the stack as a whole and just add a whole new level of complexity to maintaining MING.
For example, when working on https://gitlab.com/windfactory/hullbuoy I needed to add a bunch of nodes to node-red. So I simply appended this to the dockerfile:
RUN npm install node-red-contrib-influxdb \
    node-red-node-email \
    node-red-contrib-osc \
    node-red-node-random \
    node-red-contrib-loop-processing \
    node-red-contrib-time-range-switch \
    node-red-node-pi-gpiod

At no point would I say this should be in the MING default, but someone might think it does. At this point you're supposed to make a new layer, or simply add it yourself for your own needs rather than include it in the base layer.
Installing the node-red-influxdb-contrib makes sense, because we provide Influx.
Installing node-red-node-serialport and node-red-dashboard is sensible if it's functionality that a lot of people use and want out of the box.
But let's say Julian wanted to install node-red-node-random, prinicpally he's supposed to take MING and add that to his own fork, which is his own embedded device/package which uses that extra thing because there are an infinite amount of nodes and possibilities for things that he and his project might want or need.
At some point it would get ridiculous and should not be included in the base, you should go off and copy MING and add those things for your own purposes/project.
That said, I think is right to include all of the things Julian has mentioned since they are small and useful additions that are highly geared towards the use case of everyone using MING. I will however add a build arg to not include them. That ARG would be barebones or something of that nature, so that when building people can use docker build --build-args=barebones and remove the un-necessary componentry.
I'll submit a PR later tonight or tomorrow with these changes for review.

Also can't run apt-get install, because I want Git to see if I can get micropython.

The only container in use that is based on Debian is Jupyter. And as mentioned by Alex, you do need to run apt update the first time the container is ran before you can use apt.

Where are these /data directories?
They're volumes. Volumes have Docker documentation, https://docs.docker.com/storage/volumes/.

They're in /var/lib/docker/volumes with their relevant names as specified in the docker-compose.yml.
https://github.com/DynamicDevices/ming/blob/4c77867e654a5bb35ee6d450e5d0c6d5ca3fa8b8/docker-compose.yml#L2-L6

At no point would I say this should be in the MING default, but someone might think it does. At this point you're supposed to make a new layer, or simply add it yourself for your own needs rather than include it in the base layer.

Agreed. It's finding the line.

Installing the node-red-influxdb-contrib makes sense, because we provide Influx.

Agreed.

Installing node-red-node-serialport and node-red-dashboard is sensible if it's functionality that a lot of people use and want out of the box.

Yes. Agreed.

But let's say Julian wanted to install node-red-node-random, principally he's supposed to take MING and add that to his own fork, which is his own embedded device/package which uses that extra thing because there are an infinite amount of nodes and possibilities for things that he and his project might want or need.
At some point it would get ridiculous and should not be included in the base, you should go off and copy MING and add those things for your own purposes/project.

Agreed. We need to consider the downside of installing extra nodes. In general I would say there is very little downside to adding functionality to node-red

That said, I think is right to include all of the things Julian has mentioned since they are small and useful additions that are highly geared towards the use case of everyone using MING. I will however add a build arg to not include them. That ARG would be barebones or something of that nature, so that when building people can use docker build --build-args=barebones and remove the un-necessary componentry.

Yes that would be excellent

I'll submit a PR later tonight or tomorrow with these changes for review.

Ta!

I can see the file systems.  It's fairly obscure and su protected, here:
sudo ls /var/lib/docker/volumes/ming_jupyterlab-data/_data
To get into the jupyter image to add new stuff when you don't have root access:

docker exec -u 0 -it ming_jupyterlab_1 bash
You can then set the su password with "passwd"

We might want both "jupyter notebook" and "jupyter lab" running.
The latter is the up to date version, but the former is the more hacky because it is allowed to run arbitrary unsanitized javascript, and therefore has better interactivity and features.
eg this works in it:  https://ipyvolume.readthedocs.io/en/latest/
The interactive sliders and stuff don't work in lab.  It can be re-enabled here:
https://stackoverflow.com/questions/49542417/how-to-get-ipywidgets-working-in-jupyter-lab
I ran this before setting it up:

jupyter labextension install @jupyter-widgets/jupyterlab-manager

Then did a docker container restart ming_jupyter_1.
And now the simplest widget example works:
from ipywidgets import widgets
def f(x):
    print(x)
widgets.interact(f, x=10)

This is very good.
Now for the 3D capabilities as outlined here: https://pythreejs.readthedocs.io/en/stable/ or at https://github.com/jupyter-widgets/pythreejs

jupyter labextension install jupyter-threejs

I got the pythreejs to work, but it's way too low level and complicated to drive, in comparison to https://ipyvolume.readthedocs.io/en/latest/  There is no way scientists could ever use it.
Here's some docs:   https://github.com/maartenbreddels/ipyvolume

conda install -c conda-forge nodejs  # or some other way to have a recent node
jupyter labextension install @jupyter-widgets/jupyterlab-manager
jupyter labextension install ipyvolume
jupyter labextension install jupyter-threejs

I'm fixing meta-mono at the moment but intending to sort out sudo access for you in the Jupyter container

@goatchurchprime  What are you doing to run MING locally on your laptop? (command lines please?)

@ajlennon Why is sudo necessary inside the container?

Also if we're going to make changes like this, it makes sense at this point to have a development branch, which now that I have commit access is where I'll be putting my changes.

@goatchurchprime wants to be able to add things into the container

I've added sudo support to the Jupyter container @goatchurchprime in #26

OK you've got the node-red influxdb and serial nodes now by default #27

And the dashboard in #28

@goatchurchprime  Probably need to look at giving node-red access to the underlying serial devices as we discussed over lunch.

@ajlennon You should just be able to make the container privileged and it'll have hardware access. Alternatively you can specify devices in the docker-compose file like so https://docs.docker.com/compose/compose-file/#devices

@ajlennon You should just be able to make the container privileged and it'll have hardware access. Alternatively you can specify devices in the docker-compose file like so https://docs.docker.com/compose/compose-file/#devices

We really need to try hard to avoid granting privileged access. The problem we have is with hot-pluggable devices.

@ajlennon There is no way to do this easily otherwise. Is it really that hard to add it to the file specifically when needed? That's why I'm saying we cannot make something that is one-size fits all, there needs to be a bit of manual intervention here.
E.g make JupyterLabs privileged and accessible locally when needed.

The way Balena handles hot-plugged devices relates strongly to the base image, where they run udevadm and communicate between the host's udevd. This might be something we can add and run to each image, I have done this in OctoBalena. https://github.com/MatthewCroughan/octobalena/commit/f603b671478eeacc9010e8869159eb6d7083a944
See this for more details:
https://www.balena.io/docs/reference/base-images/base-images/#how-the-images-work-at-runtime

E.g make JupyterLabs privileged and accessible locally only on the pc it's on (<IP_ADDRESS>) when needed.

We may have to but I'd like to make sure there's absolutely no other alternative if we did as it's a security concern.

It is not a security concern if it's ran on the localhost only.  How could it possibly be? If you're the only one who can access it? Windows Explorer can do more damage..?

If you want dynamic hardware access you need udev, and that requires privilege. We can only do static hardware passthrough without privilege, as far as I have read in all the available documentation.

It's a security concern.

@ajlennon Then running Jupyter Lab on its own without docker is a security concern? Security is already improved by running it in a container at all. We're not moving backwards by giving it privilege.
Security is not the focus of the jupyterlab container, we can't move forward and do interesting things if there is a focus on security for it. Hardware access requires privilege, and by focusing too much on it we're only going to stall.

We could provide multiple compose files. One that gives hardware access and one that does not. Giving hardware access by its nature is insecure. We can't change that nature, so we shouldn't try to.

We're not moving backwards by giving it privilege.

Yes, we would be.

Security is not the focus of the jupyterlab container at least when hardware access is required. We can't move forward and do interesting things if there is a focus on security for it. Hardware access requires privilege, and by focusing too much on it we're only going to stall.

There needs to be a focus on security. Hardware access doesn't require privilege.

Seeing as Alex has blasted through a few of the to-do's, I'll make a list here of what's left to do:

[ ] Add sudo to the jupyterlab container #26
[ ] Add nodered nodes for serial support and influxdb  #27
[ ] Add nodered dashboard support #28
[ ] Figure out dynamic device discovery so that the jupyterlab can speak to devices
[ ] Add micropython kernel to jupyterlab
[ ] Add Julia kernel to jupyterlab
[ ] Add jupyterlab-manager to jupyterlab
[ ] Add pythreejs to jupyterlab
[ ] Add a recent nodejs to jupyterlab
[ ] Provide some example flows in nodered
[ ] Put all of these features in build args so they can be optionally removed at build time

We might want to use udev to kick off nc or socat to listen on a port when a USB device is inserted, then connect that port to the container
socat TCP-LISTEN:4161,fork,reuseaddr FILE:/dev/ttyUSB0,b57600,raw

e.g. https://superuser.com/questions/614494/redirect-serial-com-to-tcp-port

Im currently working on the majority of the jupyterlab issues now, in future will make separate issues so we can use the kanban board to represent what we are working on so we don't work on the same thing, though doubt that's a risk now.

None of the requested Jupyter extensions build on armv7 properly:
# Install useful JupyterLab extensions as discussed in
# https://github.com/DynamicDevices/ming/issues/25

RUN jupyter labextension install \
    @jupyter-widgets/jupyterlab-manager \
    jupyter-threejs \
    ipyvolume \
    && cat /tmp/jupyterlab-debug-*.log

The following error is returned:
[main]     An error occured.
[main]     RuntimeError: JupyterLab failed to build
[main]     See the log file for details:  /tmp/jupyterlab-debug-sc3jthqy.log
[main]     Removing intermediate container fbb8f9637bf8
Yes.. That's right. It generates a random filename in /tmp/ and doesn't give you any log information to stdout which is really annoying to debug. Works fine on x86.

Interestingly trying to install that manually does something weird and kills a basic jupyterlab container. I'm thinking it's running out of ram or something.


Bug fixes for failover

What changed?
Bug fixes for failover

Why?

compare failover version with a value
assign failover end time with the new pointer
remove the duplicated method
Add active cluster field in DomainNotActive error
Fix misuse failover notification version to failover verison

How did you test it?
Local tests

Potential risks
N/A

Coverage decreased (-0.006%) to 66.979% when pulling 4afb8ded3e6d029091ab16c9d7c4dfe583b91473 on failover_fixes into 6aca6385a1982f5661fa61ce6966fa74b7a34343 on master.


A less hacky solution to #10080

Try to deal with adding to stuff buffer between calling vungetc() and vgetc().
I haven't figured out a way to write a test either.

do_mouse() seems to be the only place where chars are added to the stuff buffer between calling vungetc() and vgetc().

I've managed to write a test.

I also wrote another test case that fails without this PR.
inoremap <LeftDrag> <LeftDrag><C-\><C-N>

Before this PR this mapping does not get out of Visual mode when dragging in Insert mode because the <C-O> from the stuff buffer is got between the <C-\> (which was put back using vungetc()) and <C-N>. Now the <C-O> is got before the <C-\>.


Allow easy toggling between sourcemaps and compiled code in js-debug

@jrieken wanted the ability to easily disable sourcemaps in js-debug, and mentioned @alexdima and @bpasero also like debugging without sourcemaps. However, changing this in the launch config via sourceMap: true/false is burdensome and makes the decision for the whole team, where it should be a user choice.
He suggested a some button (an editor action? is that too prominent?) to toggle sourcemaps on off.

The button should remember its state in the workspace between reloads
Breakpoints should always be allowed to be set in the source code, but appear in (be moved to, since BPs can only appear in one location) the compiled code when sourcemaps are disabled, and should show in the source code otherwise.

Therefore in this mode, we don't fully "disable sourcemaps", more just control where the debugger steps in
However we can optimize some things and be extra-lazy about parsing sourcemaps when "disabled"

Editor title action might be too prominent. Why not Call Stack title area action? I think extensions can not contribute to this place right now, but we could change this.
fyi @roblourens for ideas.

Why not Call Stack title area action?

Because it isn't an action that operates on call stacks but on the whole file/editor?

I think this would operate on the whole debug session, not one editor. That makes me think it would make more sense as a context menu item on the debug session. I also don't think this will be commonly used enough to have a button for it. But, it should be really obvious somehow that this mode is activated.

Also, you can maintain your own launch configs in a workspace config, I do this sometimes. But I get that it's a pain to clone our huge vscode launch.json

I like that, it could also go into the editor title at that point

@connor4312 sounds good to me.


Export urlEndpoint

Can urlEndpoint be exported? Right now I have to String() and re-parse the endpoint in order to get the URL object out for my custom transport. :-(

All Endpoint implementations could be exported. Could you make a PR for this?


Error WriteWrap.afterWrite(net) on quit

see https://sentry.iobroker.net/share/issue/1c47c20de09f416ca424a22c861e6a5b/
Error: write EPIPE
  File "net.js", line 789, col 14, in WriteWrap.afterWrite
  File "    --------------------", in null.<anonymous>
  File "/opt/iobroker/node_modules/mysql/lib/protocol/Protocol.js", line 144, col 48, in Protocol._enqueue
    sequence._callSite = sequence._callSite || new Error();
  File "/opt/iobroker/node_modules/mysql/lib/protocol/Protocol.js", line 91, col 23, in Protocol.quit
    var sequence = this._enqueue(new Sequences.Quit(options, callback));
  File "/opt/iobroker/node_modules/mysql/lib/Connection.js", line 239, col 18, in Connection.end
    this._protocol.quit(opts, wrapCallbackInDomain(this, cb));
  File "/opt/iobroker/node_modules/sql-client/lib/connection-factory.js", line 22, col 20, in MySQLConnectionFactory.close_connection
    connection.end();
  File "/opt/iobroker/node_modules/sql-client/lib/sql-client.js", line 51, col 29, in SQLClient.disconnect
    return this.factory.close_connection(this.connection, (err) => {
  File "/opt/iobroker/node_modules/sql-client/lib/sql-client-pool.js", line 42, col 25, in MySQLClientPool.destroy
    return client.disconnect(callback);
  File "/opt/iobroker/node_modules/sql-client/lib/sql-client-pool.js", line 234, col 31, in passivate
    return this.destroy(client, callback);
  File "/opt/iobroker/node_modules/sql-client/lib/sql-client-pool.js", line 37, col 16, in MySQLClientPool.passivate
    return callback(null, client);
  File "/opt/iobroker/node_modules/sql-client/lib/sql-client-pool.js", line 227, col 25, in MySQLClientPool.return
    return this.passivate(client, (err, client) => {
  File "/opt/iobroker/node_modules/iobroker.sql/main.js", line 246, col 43, in returnClientToPool
    return clientPool && clientPool.return(client);

And it seems that also the registered error eent handler do not get that exception

The error write EPIPE is an error from Node.js core. It means that this module was unable to write the QUIT sequence to the server due to the TCP connection most likely ending at some point. The error will come back as the first argument to the callback provided to the connection.end method. You can read more about error handling and this module here: https://github.com/mysqljs/mysql#error-handling

Ok, thank you, I created an issue in the library that also has it's part :-)


Proposal: fix of Docker image

Adding author (@garciagenrique feel free to comment/change)
Replacing spawn of the notebook with the CMD ["start-notebook.py"] command.

lgtm, but if we use start-notebook.py the user needs to see the token. Could you please remove the -d from the README file?
https://github.com/vre-hub/reana-jupyterlab-extension/blob/da287f5d46453d0d44a20f0f373f2bb9a4915de9/README.md?plain=1#L34

Let's remove the detach option and update doc then :)


Maven publishment?

Hi! I found this great project but noticed that you didn;t upload it to maven central or any repo
Will you upload to a repository? Becuz I don't want to put the source code into my project directly.

The project is not on maven central, but you can use it via jitpack. See the example usage at https://github.com/vonox7/golfcoder/blob/main/build.gradle.kts#L26 and https://github.com/vonox7/golfcoder/blob/main/build.gradle.kts#L50-L51:
repositories {
    mavenCentral()
    maven("https://jitpack.io") // Needed for katerbase
}

dependencies {
    ...
    implementation("com.github.studoverse:katerbase:feb131944cef43e6accd101d3b2579e7a0b171b7")
    implementation("org.mongodb:mongodb-driver-sync:4.9.0")
}


[SyncWith:zkllvm#530] Code cleanup v1

Partially closing this issue:

#180

Here are some code refactoring and cleanups.
It may be more handy to review changes with commits one by one. Each commit has some explanations of changes made.

Merge after:

#183


Feature coldpool

These changes will implement the cold pool index within the mom6 index module that will generate the index based on the current mask data that we have. These changes also include methods to generate and retrieve the mask based on GLORYS data. This is intended to be a temporary change until we have access to the mask data used in the Ross et al., 2023 paper.

@chiaweh2 The cold pool index logic has passed my tests and I believe it's good to go to the development branch. Please take a look and let me know if you think anything needs to change. Thanks!


Implement xenstore permissions

This PR mainly introduces of xenstore permissions feature. The feature is needed to allow or deny access between domains to process some operations with xenstore entries. In additionally to permissions support, also implemented the following:

Removing of entries that were created for domain or by domain in case of destruction;
Helper functions to create entries and set permission in one action;
Moved initialize_xenstore function to appropriate place;
Minor fix: /tool/xenstored was created every time on domain creation, that is not correct.

Please note, that the exact checking permissions were implemented in the last commit in order not to make one large commit and save code consistency between commits.

As I see, CI checkpatch runs for the previous commits and prints warnings not related to PR. Also, PR has 2 expected warnings (more than 100 columns in one line), but it's because of forbidding to separate log messages.

@lorc @firscity thank you for your close review.
I released a new version of PR, so, please take a look.
All your comments are fixed.
The new PR contains the following changes compared to the previous:

Changed domid type in xs_permissions structure to domid_t;
Changed perms type in xs_permissions structure enum xs_perm;
Removed double initialization of iteration pointer in string_to_perms function;
Removed redundant newline;
Implemented helper function for safe conversion from string to uint16_t to avoid using atoi function;
Add missing incrementation of j variable in string_to_perms;
Changed some function namings to more appropriate;
Moved back initialize_xenstore function and just refactored it to use the new API ;
Moved deinitialize_domain_xenstore to xen-dom-mgmt.c to separate entities of xenstore client and server.

@lorc thank you again for your close review and rechecking the specification. I apologize for misunderstanding about the first permissions entry meaning - it was really uncommon thing.
Please, take a look at the new version of PR. In comparison with the previous the following changes are made:

Reworked checking permissions: taking into account the value of the first permission entry, which means permissions for other domains that are not explicitly defined;
Changed custom function for string to integer conversion to strtoul;
Changed the namings of the functions to show that we are working with multiple strings separated by terminating null;
Added some useful comments;
Implemented helper function to check if domain is owner. With this function, we can omit using XS_PERM_OWNER, which was removed at all.

@lorc thank you for your review.
All your comments are fixed.
In comparison with the previous PR version the following changes are made:

Removed redundant comment;
Removed redundant comma;
Introduced a new function key_to_entry_check_perm and left key_to_entry as it was;
Added @lorc RB.

@firscity thank you for your close review. Added acked-by


Allow setting a custom KEEPALIVE_TIMEOUT

When the KEEPALIVE_TIMEOUT environment variable is set, we'll replace
the default 5 seconds timeout with whatever value was found in there.
This may eventually be exposed to end-users, so we validate that it's an
integer.

This is similar to https://github.com/aptible/docker-nginx/pull/77 with a few differences:

It works for both HTTP and HTTPS
It doesn't allow a timeout that's not an integer
It has a few basic tests

cc @fancyremarker @kunwarbhatia

Note: I'm not sure whether we'll actually expose this variable to customers yet. AFAICT there was only demand for it on ELB endpoints to reduce latency, but since ELB endpoints are deprecated now, that may not be very relevant.
However, it looks like we're going to want to use this internally to increase the keep alive timeout between an ALB and the downstream Nginx proxies (since otherwise we may get occasional 502's when the ALB attempts to reuse a connection that Nginx had closed).


Moretwins

add VICReg class subclassing the general parent class Twins for siamese networks.
Other losses could be easily implemented, e.g. SimCLR or the like.
Possible generalization: look into student-teacher networks such as BYOL

N.B: Twins signature has changed: [n, 2, d] -> [n, 2, d'] for consistency with the Module class


the trait bound Option<i32>: ToSchema is not satisfied

Hi. Im trying out the new generic support for 0.1.0-rc.0 and I am running into issues updating my code.
By using the following code, I am getting an error when I'm trying to create a struct
pub type optionTransaction =  IdentifiableTransactionBase<Transaction, Option<Uuid>>;
pub type nonOptionTransaction =   IdentifiableTransactionBase<Transaction, Uuid>;

#[derive(Clone, Debug, Serialize, Deserialize, ToSchema)]
pub struct IdentifiableTransactionBase<B: ToSchema, I: ToSchema> {
    /// Id representing the full transaction.
    pub transaction_id: I,

    #[serde(flatten)]
    pub base: B,
}

the trait bound 'uuid::Uuid: ToSchema' is not satisfied
This was not an issue before when using aliases. I apologise if this was already discussed somewhere and I have missed it :)

It might be a problem in the the rc itself. I tried the following in the master in to utoipa-config/config-test-crate/ and it works no errors:
pub type optionTransaction =  IdentifiableTransactionBase<Transaction, Option<Uuid>>;
pub type nonOptionTransaction =   IdentifiableTransactionBase<Transaction, Uuid>;

#[derive(Clone, Debug, Serialize, Deserialize, ToSchema)]
pub struct IdentifiableTransactionBase<B: ToSchema, I: ToSchema> {
    /// Id representing the full transaction.
    pub transaction_id: I,

    #[serde(flatten)]
    pub base: B,
}

It might be issue that is fixed here https://github.com/juhaku/utoipa/pull/1079

Hi. Thanks for the swift reply.
I actually tried master as well but got bombarded with 1000s of errors :D
Will atempt again and report back the results.

:sweat_smile: heh, yeah there are plenty of changes. Though eventually I will write some sort of migration guide about the fundamental changes once I get that far.

Yeah can confirm its a problem with the rc.
Created minimal reproducible example
[package]
name = "utoipatest"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
uuid = { version = "1.10.0", features = ["serde", "v4"] }
utoipa = { version = "5.0.0-rc.0", features = ["axum_extras", "time", "uuid", "decimal_float", "config"] }

use utoipa::ToSchema;
use uuid::Uuid;

pub type optionTransaction = IdentifiableTransactionBase<i32, Option<Uuid>>;
pub type nonOptionTransaction = IdentifiableTransactionBase<i32, Uuid>;

#[derive(Clone, Debug, ToSchema)]
pub struct IdentifiableTransactionBase<B: ToSchema, I: ToSchema> {
    /// Id representing the full transaction.
    pub transaction_id: I,

    pub base: B,
}

fn main() {
    println!("Hello, world!");
    let trans = nonOptionTransaction {
        transaction_id: Uuid::new_v4(),
        base: 1,
    };
}

And getting
error[E0277]: the trait bound `uuid::Uuid: ToSchema` is not satisfied
  --> src/main.rs:17:17
   |
17 |     let trans = nonOptionTransaction {
   |                 ^^^^^^^^^^^^^^^^^^^^ the trait `ToSchema` is not implemented for `uuid::Uuid`
   |
   = help: the following other types implement trait `ToSchema`:
             bool
             char
             isize
             i8
             i16
             i32
             i64
             i128
           and 13 others
note: required by a bound in `IdentifiableTransactionBase`
  --> src/main.rs:8:56
   |
8  | pub struct IdentifiableTransactionBase<B: ToSchema, I: ToSchema> {
   |                                                        ^^^^^^^^ required by this bound in `IdentifiableTransactionBase`

But with master
utoipa = { git = "https://github.com/juhaku/utoipa.git", features = ["axum_extras", "time", "uuid", "decimal_float", "config"] }
and changing to pub struct IdentifiableTransactionBase<B, I> {
the code compiles fine :)
Apologies for the issue raised.
Will atempt to migrate my project to master and see if I get any luck. Thanks for the great work again!

No worries, great that you got it working.

Hello.
I am back and re-opening the issue. I hope thats okay, if not, I will close and create a new issue.
I am experiancing the same issue as before, but this time, not when I try to create an object, but rather when I try to define and enum using that type. This is happening in master.
Here is a reproducible example
Catgo.toml
[package]
name = "utoipatest"
version = "0.1.0"
edition = "2021"

[dependencies]
utoipa = {git = "https://github.com/juhaku/utoipa.git" }

main.rs
use utoipa::ToSchema;

pub type OptionFoo = FooStruct<Option<i32>>;

#[derive(ToSchema)]
pub struct FooStruct<B> {
    pub foo: B,
}

#[derive(ToSchema)]
pub enum FoosEnum {
    FooEnumThing(OptionFoo),
}

fn main() {
    println!("Hello, world!");
}

Error
error[E0277]: the trait bound `Option<i32>: ToSchema` is not satisfied
  --> src/main.rs:12:18
   |
12 |     FooEnumThing(OptionFoo),
   |                  ^^^^^^^^^ the trait `ToSchema` is not implemented for `Option<i32>`, which is required by `FooStruct<Option<i32>>: ToSchema`
   |
   = help: the following other types implement trait `ToSchema`:
             bool
             char
             isize
             i8
             i16
             i32
             i64
             i128
           and 14 others
note: required for `FooStruct<Option<i32>>` to implement `ToSchema`
  --> src/main.rs:5:10
   |
5  | #[derive(ToSchema)]
   |          ^^^^^^^^ unsatisfied trait bound introduced in this `derive` macro
6  | pub struct FooStruct<B> {
   |            ^^^^^^^^^^^^
   = note: this error originates in the derive macro `ToSchema` (in Nightly builds, run with -Z macro-backtrace for more info)

Hope this helps :)

I am back and re-opening the issue. I hope thats okay

Yeah it is alright. And it does help indeed, :slightly_smiling_face: It is always good to have something concrete to reproduces or at least to point out the thing that does not wok to help with debugging.
I need to further investigate this a bit, I belive that I have missed the alias TypeTree from some places like in the enum variant processing. I'll try to look into this as next thing on the list.

This is actually different issue and is not releated to the original aliases. This is issue rises because of limitations of current generics implementation which does not implement ToSchema for container types like Option<T>, Vec<T>, etc. because they are automatically inlined when they are used as types for fields or variants. The current solution does not actually know how to correctly handle those container types when they are used as generic arguments for types.
This actually is not a small fix, but rather some real engineering effort need to be put for this in order to get the types correct (if possible). I'll try to see what can be done, nonetheless.

There is fix for this coming up in #1107 but with few limitations. tuples, slices and arrays cannot be used as generic arguments. This would need some internal changes that need to be addressed in future. From now on concrete types should work as well.
Note to above, for type aliases to correctly generate the schema they need to be registered in build.rs with utoipa-config. If not then the underlying type will not be correct schema even if it manages to generate one.

pub type OptionFoo = FooStruct<Option<i32>>;

#[derive(ToSchema)]
pub struct FooStruct<B> {
   pub foo: B,
}

#[derive(ToSchema)]
pub enum FoosEnum {
   FooEnumThing(OptionFoo),
}

This is amazing! Im always amazed how responsive you are to the community.
Is there a way to buy you a coffee? Did not find any link for that :)
Either way, it appears I still have a similar problem. This time with UUID again as in initial query. Probably should have tested this before hand and raised it as well, so apologies for the back and forth.
Here is a reproducable example
[package]
name = "utoipatest"
version = "0.1.0"
edition = "2021"

[dependencies]
utoipa = {git = "https://github.com/juhaku/utoipa.git" }
uuid = { version = "1.10.0", features = ["v4"] }

use utoipa::ToSchema;
use uuid::Uuid;

pub type OptionFoo = FooStruct<Uuid>;

#[derive(ToSchema)]
pub struct FooStruct<B> {
    pub foo: B,
}

#[derive(ToSchema)]
pub enum FoosEnum {
    FooEnumThing(OptionFoo),
}

fn main() {
    // This is okay now. It did not work inittially when I raised the issue.
    let a = OptionFoo { foo: uuid::Uuid::new_v4() };
}

error[E0277]: the trait bound `Uuid: ToSchema` is not satisfied
  --> src/main.rs:13:18
   |
13 |     FooEnumThing(OptionFoo),
   |                  ^^^^^^^^^ the trait `ToSchema` is not implemented for `Uuid`, which is required by `FooStruct<Uuid>: ToSchema`
   |
   = help: the following other types implement trait `ToSchema`:
             bool
             char
             isize
             i8
             i16
             i32
             i64
             i128
           and 27 others
note: required for `FooStruct<Uuid>` to implement `ToSchema`
  --> src/main.rs:6:10
   |
6  | #[derive(ToSchema)]
   |          ^^^^^^^^ unsatisfied trait bound introduced in this `derive` macro
7  | pub struct FooStruct<B> {
   |            ^^^^^^^^^^^^
   = note: this error originates in the derive macro `ToSchema` (in Nightly builds, run with -Z macro-backtrace for more info)

You need the uuid feature flag for utiopa most likely.
[dependencies]
utoipa = {git = "https://github.com/juhaku/utoipa.git", feature = ["uuid"] }
uuid = { version = "1.10.0", features = ["v4"] }

This is amazing! Im always amazed how responsive you are to the community.

It just that I have now plenty of time :smile: I should really set the github sponsors up, I just haven't as of yet. But thanks anyways :slightly_smiling_face:

You need the uuid feature flag for utiopa most likely.
Unfortunately, even after adding uuid feature flag it still has the same error :(

I have it in my main project and got the same error, just forgot to include it in the example :D

Ah, right, okay, now I know what is going on. This hit again another limitation of the current generics. The issue arises now in cases where a certain type should be treated as a primitive type but is an external type actually. That is, in general all schemas should implement ToSchema and when it comes to Uuid utoipa will check whether the type is a primitive type and tokenize it. If uuid feature is enabled this check will be true in case of Uuid type. However this only works with fields of a struct or enum variants but no on generic arguments since the generics is quite a new thing this wasn't  considered there. This still needs some experimenting around possibilities.
The quick "fix" for your issue is to wrap it to a new type and use that as a generic argument.
#[derive(ToSchema)]
struct MyUuid(Uuid);

#[derive(ToSchema)]
pub struct FooStruct<B> {
    pub foo: B,
}

let a = FooStruct {
  foo: MyUuid(Uuid::new_v4())
}

@EinarasGar This PR #1110 will enhance the generics with support for allowing known third party types to be used as a generic argument fixing the issue you are encountering with.

Thanks again for taking a look! Very fast.
Unfortunately I still see the same error even with your provided example. Perhaps im missing something?
I verified that I am on latest commit fc22ddf
use utoipa::ToSchema;

#[derive(ToSchema)]
pub struct High<T> {
    high: T,
}

#[derive(ToSchema)]
pub struct HighUuid(High<Option<uuid::Uuid>>);

fn main() {

}

[package]
name = "utoipatest"
version = "0.1.0"
edition = "2021"

[dependencies]
utoipa = {git = "https://github.com/juhaku/utoipa.git", feature = ["uuid"] }
uuid = { version = "1.10.0", features = ["v4"] }

warning: unused manifest key: dependencies.utoipa.feature
    Checking utoipatest v0.1.0 (/workspace/test/utoipatest)
error[E0277]: the trait bound `Uuid: ToSchema` is not satisfied
 --> src/main.rs:9:21
  |
9 | pub struct HighUuid(High<Option<uuid::Uuid>>);
  |                     ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `ToSchema` is not implemented for `Uuid`, which is required by `High<Option<Uuid>>: ToSchema`
  |
  = help: the following other types implement trait `ToSchema`:
            bool
            char
            isize
            i8
            i16
            i32
            i64
            i128
          and 27 others
  = note: required for `Option<Uuid>` to implement `ToSchema`
  = note: 1 redundant requirement hidden
  = note: required for `High<Option<Uuid>>` to implement `ToSchema`

error[E0277]: the trait bound `Uuid: ComposeSchema` is not satisfied
 --> src/main.rs:9:21
  |
9 | pub struct HighUuid(High<Option<uuid::Uuid>>);
  |                     ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `ComposeSchema` is not implemented for `Uuid`, which is required by `High<Option<Uuid>>: ToSchema`
  |
  = help: the following other types implement trait `ComposeSchema`:
            High<T>
            HighUuid
            Box<T>
            Cow<'a, T>
            HashMap<K, T>
            BTreeMap<K, T>
            HashSet<K>
            BTreeSet<K>
          and 7 others
  = note: required for `Uuid` to implement `PartialSchema`
  = note: required for `Option<Uuid>` to implement `ComposeSchema`
  = note: 1 redundant requirement hidden
  = note: required for `Option<Uuid>` to implement `PartialSchema`
  = note: required for `Option<Uuid>` to implement `ToSchema`
  = note: 1 redundant requirement hidden
  = note: required for `High<Option<Uuid>>` to implement `ToSchema`

error[E0277]: the trait bound `Uuid: ToSchema` is not satisfied
 --> src/main.rs:9:26
  |
9 | pub struct HighUuid(High<Option<uuid::Uuid>>);
  |                          ^^^^^^^^^^^^^^^^^^ the trait `ToSchema` is not implemented for `Uuid`, which is required by `Option<Uuid>: ToSchema`
  |
  = help: the following other types implement trait `ToSchema`:
            bool
            char
            isize
            i8
            i16
            i32
            i64
            i128
          and 27 others
  = note: required for `Option<Uuid>` to implement `ToSchema`

error[E0277]: the trait bound `Uuid: ComposeSchema` is not satisfied
   --> src/main.rs:9:26
    |
9   | pub struct HighUuid(High<Option<uuid::Uuid>>);
    |                          ^^^^^^^^^^^^^^^^^^ the trait `ComposeSchema` is not implemented for `Uuid`, which is required by `Option<Uuid>: PartialSchema`
    |
    = help: the following other types implement trait `ComposeSchema`:
              High<T>
              HighUuid
              Box<T>
              Cow<'a, T>
              HashMap<K, T>
              BTreeMap<K, T>
              HashSet<K>
              BTreeSet<K>
            and 7 others
    = note: required for `Uuid` to implement `PartialSchema`
    = note: required for `Option<Uuid>` to implement `ComposeSchema`
    = note: 1 redundant requirement hidden
    = note: required for `Option<Uuid>` to implement `PartialSchema`
note: required by a bound in `name`
   --> /home/vscode/.cargo/git/checkouts/utoipa-797003a51754f4fd/fc22ddf/utoipa/src/lib.rs:403:21
    |
403 | pub trait ToSchema: PartialSchema {
    |                     ^^^^^^^^^^^^^ required by this bound in `ToSchema::name`
...
434 |     fn name() -> Cow<'static, str> {
    |        ---- required by a bound in this associated function

error[E0277]: the trait bound `Uuid: ToSchema` is not satisfied
 --> src/main.rs:9:33
  |
9 | pub struct HighUuid(High<Option<uuid::Uuid>>);
  |                                 ^^^^^^^^^^ the trait `ToSchema` is not implemented for `Uuid`
  |
  = help: the following other types implement trait `ToSchema`:
            bool
            char
            isize
            i8
            i16
            i32
            i64
            i128
          and 27 others

error[E0277]: the trait bound `Uuid: ComposeSchema` is not satisfied
   --> src/main.rs:9:33
    |
9   | pub struct HighUuid(High<Option<uuid::Uuid>>);
    |                                 ^^^^^^^^^^ the trait `ComposeSchema` is not implemented for `Uuid`, which is required by `Uuid: PartialSchema`
    |
    = help: the following other types implement trait `ComposeSchema`:
              High<T>
              HighUuid
              Box<T>
              Cow<'a, T>
              HashMap<K, T>
              BTreeMap<K, T>
              HashSet<K>
              BTreeSet<K>
            and 7 others
    = note: required for `Uuid` to implement `PartialSchema`
note: required by a bound in `name`
   --> /home/vscode/.cargo/git/checkouts/utoipa-797003a51754f4fd/fc22ddf/utoipa/src/lib.rs:403:21
    |
403 | pub trait ToSchema: PartialSchema {
    |                     ^^^^^^^^^^^^^ required by this bound in `ToSchema::name`
...
434 |     fn name() -> Cow<'static, str> {
    |        ---- required by a bound in this associated function

For more information about this error, try `rustc --explain E0277`.
error: could not compile `utoipatest` (bin "utoipatest") due to 12 previous errors

That's weird, like I created a test project and setup it and it is running perfect:

Perhaps cargo clean will help? :thinking:

I also tried cargo clean and even peeked into the code itself to verify that your changes are in.
Let me try to spin up a new dev container and reproduce it again :D

Ran a new docker dev container with rust pre-installed, still same error :( Even recreated your screenshot 1:1

devcontainer.json
{
	"name": "Rust",
	"image": "mcr.microsoft.com/devcontainers/rust:1-1-bullseye"
}

:open_mouth: That is so real, at least the one feature is wrong but not sure whether that actually has any impact.

Yeah just noticed and fixed axum_extras feature, but still same

Yeah, This is what it suppose to generate if you expand the ToSchema macro on that HighUuid type.
impl utoipa::__dev::ComposeSchema for HighUuid {
    fn compose(
        mut generics: Vec<utoipa::openapi::RefOr<utoipa::openapi::schema::Schema>>,
    ) -> utoipa::openapi::RefOr<utoipa::openapi::schema::Schema> {
        utoipa::openapi::schema::RefBuilder::new()
            .ref_location_from_schema_name(std::borrow::Cow::Owned(format!(
                "{}_{}",
                <High<Option<String>> as utoipa::ToSchema>::name(),
                std::borrow::Cow::<String>::Owned(
                    [std::borrow::Cow::Owned(format!(
                        "{}_{}",
                        <Option<String> as utoipa::ToSchema>::name(),
                        std::borrow::Cow::<String>::Owned(
                            [<String as utoipa::ToSchema>::name(),].to_vec().join("_")
                        )
                    )),]
                    .to_vec()
                    .join("_")
                )
            )))
            .into()
    }
}
impl utoipa::ToSchema for HighUuid {
    fn name() -> std::borrow::Cow<'static, str> {
        std::borrow::Cow::Borrowed("HighUuid")
    }
}
impl utoipa::__dev::SchemaReferences for HighUuid {
    fn schemas(
        schemas: &mut Vec<(
            String,
            utoipa::openapi::RefOr<utoipa::openapi::schema::Schema>,
        )>,
    ) {
        schemas.extend([(
            String::from(std::borrow::Cow::Owned(format!(
                "{}_{}",
                <High<Option<String>> as utoipa::ToSchema>::name(),
                std::borrow::Cow::<String>::Owned(
                    [std::borrow::Cow::Owned(format!(
                        "{}_{}",
                        <Option<String> as utoipa::ToSchema>::name(),
                        std::borrow::Cow::<String>::Owned(
                            [<String as utoipa::ToSchema>::name(),].to_vec().join("_")
                        )
                    )),]
                    .to_vec()
                    .join("_")
                )
            ))),
            <High<Option<String>> as utoipa::__dev::ComposeSchema>::compose(
                [<Option<String> as utoipa::__dev::ComposeSchema>::compose(
                    [<String as utoipa::PartialSchema>::schema()].to_vec(),
                )]
                .to_vec(),
            ),
        )]);
        <High<Option<String>> as utoipa::__dev::SchemaReferences>::schemas(schemas);
    }
}

Does the devcontainers use local cache for the the crates.io index or is it global on your system? If it is global you could try to remove the record from the caced index on your machine.
registry/src/index.crates.io-6f17d22bba15001f
❯ ls -l | grep utoipa
drwxr-xr-x - juha 18 Apr 00:49 utoipa-4.0.0
drwxr-xr-x - juha 29 Aug 17:08 utoipa-4.2.3
drwxr-xr-x - juha 18 Apr 00:49 utoipa-gen-4.0.0
drwxr-xr-x - juha 29 Aug 17:08 utoipa-gen-4.3.0
drwxr-xr-x - juha 18 Apr 00:49 utoipa-swagger-ui-4.0.0
drwxr-xr-x - juha  8 Sep 10:59 utoipa-swagger-ui-7.1.0
registry/src/index.crates.io-6f17d22bba15001f
❯ cd ../../cache
.cargo/registry/cache
❯ ls -l | grep utoipa
.cargo/registry/cache
❯ cd index.crates.io-6f17d22bba15001f
registry/cache/index.crates.io-6f17d22bba15001f
❯ ls -l | grep utoipa
.rw-r--r--   54k juha 18 Apr 00:49 utoipa-4.0.0.crate
.rw-r--r--   55k juha 29 Aug 17:08 utoipa-4.2.3.crate
.rw-r--r--  141k juha 18 Apr 00:49 utoipa-gen-4.0.0.crate
.rw-r--r--  143k juha 29 Aug 17:08 utoipa-gen-4.3.0.crate
.rw-r--r--  4,4M juha 18 Apr 00:49 utoipa-swagger-ui-4.0.0.crate
.rw-r--r--   26k juha  8 Sep 10:59 utoipa-swagger-ui-7.1.0.crate
registry/cache/index.crates.io-6f17d22bba15001f

I fount the issue.
Im too new to rust to understand why, but my cargo.lock was like this
[[package]]
name = "utoipa-gen"
version = "5.0.0-rc.0"
source = "git+https://github.com/juhaku/utoipa.git?branch=master#fc22ddfd22591aa2dd110e65a166f9391df23171"
dependencies = [
 "proc-macro2",
 "quote",
 "syn",
]

Adding utoipa-gen = { git = "https://github.com/juhaku/utoipa.git", branch="master", features = ["regex","uuid"] } to dependencies fixed it

Yeah this is odd. Apologies for all the invested time.
The selling point of dev containers is that its supposed to be a fresh reproducible environment, so thats why I tried to use it.
Either way, still seems to be a problem with Enums, even though it works fine with structs

But at this point, I think you're better off spending your time on other tasks :D Already invested too much time with me

Actually it works, the issue is that Rust type alias you are using there. If you are using type aliases you need to register them with utoipa-config for utoipa. If you give the High<Uuid> directly as a value for FooEnumThing variant it will work as well.

See the more about the config https://github.com/juhaku/utoipa/tree/master/utoipa-config
https://github.com/juhaku/utoipa/tree/master/utoipa-config/config-test-crate

Ahhh I see. Makes sense!
Well that solves it. I think its the last remaining thing I had left for my migration.
Thank you for the 10th time. You should really consider adding sponsor button somewhere in your github :)

Happy to help, haha :laughing: perhaps I should.


[readme] add screenshots

Diff appears broken but should work after merge
Preview

thank you @ReadmeCritic :)


Adds X-Forward-For HTTP header support to Gunicorn

In order to get the "real" IP of an HTTP client, and not the upstream reverse proxy's IP, Gunicorn needs to support the X-Forward-For HTTP header.
For more details, see:

http://gunicorn-docs.readthedocs.org/en/latest/settings.html#forwarded-allow-ips
http://gunicorn-docs.readthedocs.org/en/latest/deploy.html

@awolfe76, could you check this one out?  Just a one-line config change.

:+1: