Retries doesn't help against timeouts on downloads

Package Name: azure-storage-blob
Package Version: 12.19.1
Operating System: Fedora Workstation (39)
Python Version: 3.12.2

Describe the bug
When experiencing timeouts while downloading a blob with Azure Python SDK the whole download will fail when the network is brought back to normal even if there are retries left for the retry_policy.
I have not been able to reproduce this with uploads.
To Reproduce
See https://github.com/mattiasb/blob-issue for more details on how to reproduce (including an example script).
Steps to reproduce the behavior:

Start downloading a sufficiently large blob using this script.
After a little while induce a really large artificial network latency to your network device. For example with this script.
Wait for two rounds of retries to trigger
Reset the network to normal
Watch the download fail even though the retry policy has retries left and the network is back to normal.

Expected behavior
I would expect the retry policy to trigger also on timeouts, such that when I returned the network to normal a later set of retries would let the script finish downloading the file.
Additional context
Again. The best is to see what I do at https://github.com/mattiasb/blob-issue. But I recorded some logs as well which might be worth looking at.

Thanks for the feedback @mattiasb . We will investigate and get back to you asap.

Hi @mattiasb, sorry for the delay. I believe what you are encountering is that there is actually a separate (naive and undocumented) retry policy for larger downloads that use response streaming. In the internal download code there is a retry loop when trying to stream the response body of each chunk. This retry policy has its own retry count and does not interact with the global retry policy. From the stacktrace in your logs, you can see the error is coming from here.
This retry policy is a side-effect of the way response streaming in handled by azure-core and the way the SDK is setup to download the blob in chunks. Streaming the body from the server comes after the initial HTTP response and therefore is not run through the normal transport pipeline. Without this, any error while streaming would cause the whole download to fail.
So, what is likely happening is you are exhausting this retry policy and the failure is bubbling up but is not an error that is handled by the global retry policy and therefore it is possible your request can fail without all the retry attempts being exhausted.
We realize this is not the most ideal implementation, especially since the internal retry count cannot be adjusted and we would like to improve this in the future, but for now this is the likely cause for what you are seeing. In real-world applications, we have found this policy to be effective in mitigating random network interruptions so you may not run into any issues in a production environment. Or for now, you can also consider implementing your own retry over the download call.

Or for now, you can also consider
implementing your own retry over the
download call.

As a data point, this is what we have done as we do experience issues in production code.


[5.7] Soften new unavailable conformance diagnostics

Cherry-picks #42528 to release/5.7:

In https://github.com/apple/swift/pull/41054, we fixed an oversight which caused us to not notice when a user erased a concrete type to an existential using an unavailable conformance. Unfortunately, this is source-breaking and needs to be reduced to a warning in Swift 5 mode unless the user opts in.
This PR also includes an enhancement to DeclContext dumping that helps when locating imported declarations; I'm not sure this code is actually valid in its present form, and might remove it if it's a problem.
Fixes rdar://91940820.

@swift-ci please test


add tcp route

Add AppNet Tcp Route resource to rGraph

This mostly looks good -- but seems like we have to get the other changes in first.

yes, that's why this PR is marked as wip. It is dependent on PlaceholderType and ActionUpdate.

This mostly looks good -- but seems like we have to get the other changes in first.

PlaceholderType PR was merged, I applied those changes. It is now ready to review

/ok-to-test

cc/ @mag-kol


Fix/mip 45/mip engine api validator enabling

Changelog:

Refactored the controller api folder structure.
Enabled the controller api validation.
Added unit tests for the validate_algorithm method.
Added integration tests for the algorithms endpoints.
Bug fixes and other small improvements.

Codecov Report

Merging #96 (3c2d926) into master (6de80b4) will increase coverage by 14.11%.
The diff coverage is 67.46%.

@@             Coverage Diff             @@
##           master      #96       +/-   ##
===========================================
+ Coverage   21.72%   35.84%   +14.11%
===========================================
  Files          54       58        +4
  Lines        2937     3005       +68
===========================================
+ Hits          638     1077      +439
+ Misses       2299     1928      -371

Impacted Files
Coverage Δ

...controller/algorithm_executor/AlgorithmExecutor.py
0.00% <0.00%> (ø)

mipengine/controller/api/AlgorithmRequestDTO.py
100.00% <ø> (ø)

...gine/controller/api/AlgorithmSpecificationsDTOs.py
0.00% <0.00%> (ø)

mipengine/controller/api/algorithms_endpoint.py
0.00% <0.00%> (ø)

mipengine/controller/api/app.py
0.00% <0.00%> (ø)

mipengine/controller/api/error_handlers.py
0.00% <0.00%> (ø)

test_post_request.py
0.00% <0.00%> (ø)

tests/integration_tests/test_get_algorithms.py
0.00% <0.00%> (ø)

tests/integration_tests/test_post_algorithms.py
0.00% <0.00%> (ø)

mipengine/controller/algorithms_specifications.py
92.20% <80.00%> (+92.20%)
:arrow_up:

... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6de80b4...3c2d926. Read the comment docs.

@ThanKarab Would it be possible to remove all references to "cross-validation". It is stated in many places and it refers to a mechanism we have designed yet. Having unused code makes reading and understanding the code more difficult.

@apmariglis crossvalidation is removed.


regression: Pressable without functional style or children will not call onPress() on select events

Description
Observed this regression in RNTV 0.74.2-0, in a Pressable with neither a functional form of style, or of children. The regression is due to https://github.com/facebook/react-native/pull/44615 (https://github.com/react-native-tvos/react-native-tvos/commit/3f71fa8c61087d34106ac5a1ad435eab8ba20e72). Specifically, the changes in lines 387 and 392 should be reverted in order to fix the issue.
Steps to reproduce
Test case to reproduce the issue: https://github.com/douglowder/PressableExample/blob/2aab5be36424ed02a702e32c7d1f42b975ba7332/App.tsx#L80
React Native Version
0.74.2-0
Affected Platforms
Runtime - iOS
Output of npx react-native info
RNTV 0.74.2-0

Stacktrace or Logs
No crasher

Reproducer
https://github.com/douglowder/PressableExample/blob/2aab5be36424ed02a702e32c7d1f42b975ba7332/App.tsx#L80
Screenshots and Videos
No response

@Zahoq FYI

damn, you're right, thank you for pointing that out 🙏 Do we have to verify focused in part below?
      if (focused && evt.eventType === 'select') {
        // $FlowFixMe[incompatible-exact]
        onPress && onPress(evt);
      }
      // $FlowFixMe[prop-missing]
      if (focused && evt.eventType === 'longSelect') {
        // $FlowFixMe[incompatible-exact]
        onLongPress && onLongPress(evt);
      }

Pressable is called for that particular viewTag, so pressableTVFocusEventHandler shouldn't be called for any other Pressable

@Zahoq I'll test your proposed fix as well

Thanks @Zahoq I tested your proposed fix and it works fine -- merging 😄

@douglowder awesome! thanks for info! 🙌

Fixed in 0.74.3-0.


Feature: Add an option for image size

Sources
Danbooru, Gelbooru, Konachan.com (NSFW), Konachan.net (SFW), Lolibooru, Lolicon, Pixiv, Safebooru, SankakuComplex, Yande.re
Describe the problem related to the feature request
In some cases, downloading images for too long causes timeouts on the c-side.
Describe the solution you'd like
Add an option for image size (if the source supports it).
Describe alternatives you've considered
No response
Additional context
No response

Type
Define the image size type as follows:

original
large
medium
square

example:
export interface ImageSize {
  original: string
  large: string
  medium: string
  square: string
}

Behavior

Managed and returned by the image source
Starting with 'original' and fallback from maximum size

example:
support(e.g. pixiv):
{
  original: 'https://i.pximg.net/[original link]',
  large: 'https://i.pximg.net/c/600x1200_90...',
  ...
}

unsupported:
{
  original: 'https://i.pximg.net/[original link]',
  large: 'https://i.pximg.net/[original link]',
  ...
}


Subregion-toggle

Looking cool 👍

See https://github.com/covidatlas/coronadatascraper/issues/616 for a possible issue.

@lazd , should this get merged into the mapupdates branch (last commits of this were from about 2 weeks ago : https://github.com/covidatlas/coronadatascraper/commits/mapupdates )?
This PR is going into that feature branch, and not into master.  There are some questions on it, but it looks like it could keep that branch moving forward.  Thoughts?

@jzohrab not sure, but we should look at this and #367 and decide if we want those features on our map, or if we should move that development to the COVIDAtlas frontend work?

Refactored this a bit and merges this into mapupdates, thanks!


Add support for idle and toggling

Added support for an idle state (grayed out, but not black; indicates that the thermostat is working, but the heater/AC engine is off, waiting for the temperature to fall/rise).

Added support for toggling the thermostat via the context menu (long press on touchscreen devices). Not the most elegant solution (since it works differently across platforms), but for phones/tablets, it's a very convenient feature.

@aronsky I agree with 1. but I'm not sure 2. is the correct approach. I'm looking into adding a "settings" icon that would allow you to change "operation_mode", but I'd rather have the thermostat go up based on temperature settings not as a "switch".

Turning the operation mode based on temperature change should be the responsibility of the HA thermostat component, IMO (I have it working via automations currently - whenever I change the temperature, if the thermostat is OFF, it switches on). And that does feel the correct way to turn it on, I agree (though in the current state, it's not exactly possible to turn it on without changing the temperature). However, if I want to turn it OFF, I'd like to have an easily accessible option to do so, and not by setting the temperature (very high for cooling, or very low for heating).
The use-case is a thermostat at home, that I actually want to turn off when leaving the house, not to keep it on and idle indefinitely.


Fix branches for catkin_virtualenv

Following from the errors in https://github.com/ros/rosdistro/pull/22252/files, these branches needed to be fixed.

Thanks for fixing this up.


Facebook Login Button

Why are we checking for Facebook auth when the page is rendered? Shouldn't we prompt the user to login with a FB popup?
Also how are updating the state in the "App" component?

Yeah that's exactly what's happening, we still need the endpoint to accept
a user id when someone logs in
On Oct 9, 2015 7:23 PM, "Pritham"<EMAIL_ADDRESS>wrote:

Why are we checking for Facebook auth when the page is rendered?
https://github.com/Kong-Artist/social_reconnector/blob/master/web/static/js/components/Login.js#L6
Shouldn't we prompt the user to login with a FB popup?
Also how are updating the state in the "App" component
https://github.com/Kong-Artist/social_reconnector/blob/master/web/static/js/components/App.js#L10
?
—
Reply to this email directly or view it on GitHub
https://github.com/Kong-Artist/social_reconnector/issues/2#issuecomment-147009040
.

You're right it does. I was on Safari.
Added the login endpoints. Login is '/login'. You can check what the current user's token is with '/get_token'.


:construction: WIP Integrate parcel and refactor to more pure Vue components

Very preliminary non-working POC of refactor to use parcel bundler and refactor needed for integrating more front-end tooling for HUD. Still a lot of work to do but major improvements include:

Front-end environment variable build support (less API string interpolation on proxy)
More modular and maintainable component based architecture following Vue Single Page Component pattern
Easier dependency management with npm
Full support of ES6+ features w/ ES5 build support
Vue specific linter support
Theme-able and global styles with SCSS integration
Component level styles
Minified/obfuscated production builds for front-end assets
Cache invalidation on build
Multiple entrypoint builds
Maaayyybe hot module replacement and other front-end developer tool/integration compatibility

To install parcel:
npm install -g parcel-bundler

Create .env:
cp .env.example .env

To build bundle:
npm run build

Sample build files and real bundle sizes for Display, Panel, and Management and Drawer entrypoints and bundled components/css:

I'll try to update this with more info, I'm sure I'm missing some info and still a lot of work to do but the general framework is there and feedback more than welcome!

This pull request introduces 5 alerts when merging bcde54ea6bf9d8b0c016deb780e33f754862270e into 3393ec0f9cbb86d6f88086b563cdae6758ba555e - view on LGTM.com
new alerts:

3 for Unused variable, import, function or class
2 for Missing variable declaration

Comment posted by LGTM.com

wow wow wow wow wow. This is such incredible work. Beautifully linted code, modular and reusable components, concise descriptive commits. This is absolutely incredible. Looks great so far. @jaywon please be sure to lmk if there is anyway I can help!

Overall Structure
With the integration of Parcel, we now have Parcel acting as a transpliation/bundler to take development files with distinct syntax for variable interpolation at build time as well as path resolution. Upon build Parcel will do the following:

Minify HTML
Transpile Vue/ES6 into pure JavaScript files
Transpile SCSS templates into pure CSS files
Replace variables with {{{ ZAP_HUD_FILES }}} with their associated .env values as a replacement

This replaces the need for Java to interpolate these values at the time that assets are requested/served.
The following variables can now be set in development and compiled at build IF NEEDED on the client side at build time

ZAP_HUD_FILES=https://zap
ZAP_HUD_URL=https://targetdomain.com (ASSUMING STILL NEEDED IN JAVA SIDE)
ZAP_HUD_API=https://zap/api
ZAP_HUD_WS=https://zap/websocket
ZAP_HUD_TOOLS=http://zap/to/some/tool
ZAP_SHARED_SECRET=sometestsecret (ASSUMING STILL NEEDED IN JAVA SIDE)
DEV_MODE= (ASSUMING STILL NEEDED IN JAVA SIDE)
TUTORIAL_URL= (NOT SURE ABOUT THIS, DIDN'T RESEARCH
ZAP_LOCALE=en_GB

The list of transpiled files so far looks like this and are built to the dist directory under zapHomeFiles for gradle to copy over:

Ultimately I think it would be preferred for the HUD Java proxy to just serve most of the static assets as fully qualified URLs on the https://zap domain rather than be requested via the ?name= query string parameter and right now that's the MAIN blocker in terms of moving forward on the front end to test and finalize the PR.
Proposed Solution
What I would like is for the following main entrypoint(s) and associated bundled assets to be served directly from the proxy as they are requested and removing the need for them to be requested with the ?name= query string parameter and have the proxy simply serve as a proxy.

Old: https://zap/?name=display.html
New: https://zap/dist/Panel/panel.html
These HTML files are then automatically updated at bundle time to reference a fingerprinted version of the .js and .css files so we need to serve those as well with fully qualified URLs w/o query string parameters.

What I am not entirely clear on is the full extent of why the randomly generated string is needed in the URL other than some comments about detecting if the HUD is in use from a security point of view, but I don't understand the full scope of that.
It's possible that the Java proxy could detect requests for the direct static assets with a 302 redirect to a URL with the random string and then validate that way to serve the contents of the file. But i'm making suggestions without full clarity on the current solution.

Thanks @jaywon :)
I'll have a good look through your proposal and give a more detailed response, but I'll give an immediate response now.
ZAP has a plugin architecture.
The API endpoint is maintained by the core.
The callback mechanism (with the random path element) was created well before we used it in the HUD, to support faking on domain content.
The HUD is a ZAP add-on.
We currently do not provide a mechanism for add-ons to register to handle arbitrary paths on the main host:port that ZAP is listening on.
That's not to say that we cant do that, but it will require core changes, and we'll need to be very careful to make sure that the HUD (or any other add-on that uses them) does not introduce vulnerabilities into ZAP.
Would you be able to cope with the existing 'random' path elements or will it break everything you're doing?
I am wary of making significant core changes this close to the 2.8.0 release...

Not sure exactly what you mean by "randomly generated strings in the url", but in my experience this is usually there for cache-busting in-case the source-map / source file changes. This is imperative when using web-worker threads / PWA because otherwise you would have to manually clear the cache which doesn't work everywhere. Maybe I don't understand enough about the HUD yet though and you are talking about something else.

@nothingismagick when the HUD communicates with ZAP it uses URLs that have a random component.
These are the 'callback' URLs that ZAP supports for faking on domain content. They only change when you restart ZAP so should have no impact on browser caching.

The new transpiled Vue files will have their own cache busting uniquely generated id's as seen in the image above. I guess I don't understand the concept of faking on domain content...
That makes sense about the plugin architecture. I had seen that there was a specific HudFileProxy.java` class and was thinking that was something that could be modified specifically for HUD compatibility without interfering with ZAP core?
The challenge is with the bundler is that it takes something like this in development:

And converts it to something like this at build time:

The prefix is customizable at build via the --public-url attribute but is meant to be a domain/path prefix and doesn't seem to be accept a fully qualified domain and open attribute like https://zap/?name=.
I could look further into what Parcel supports in terms of control of path interpolation at build time if we can't make changes to ZAP core. It's built to be extensible, so I'll look into that in the meantime.
It's just kind of an odd state is that all the front-end build tools make an assumption that files are served in a traditional web server sense.
Also thanks for the GREAT commentary @nothingismagick! I'm trying to get this unblocked as a first priority (with limited time), but very much looking forward to addressing your feedback and unblocking you to execute on all of these ideas.

Other example is the CSS. Parcel allows us to just include <style> blocks in the .vue components with no reference in the HTML:

And converts it to this at build time with no reference at all in the development files:

These are all great improvements as it relates to theming, build time interpolation, support of ES6+ features, Vue best practices/developer experience, dependency management, and such but I'm sure we'll figure something out 😃

@jaywon so the ?name= path element is under the HUDs control :)
Its handled here: https://github.com/zaproxy/zap-hud/blob/develop/src/main/java/org/zaproxy/zap/extension/hud/HudFileProxy.java#L107
We can easily change that part, but we will need to know the difference between files and images as they are held in different directories: https://github.com/zaproxy/zap-hud/blob/develop/src/main/java/org/zaproxy/zap/extension/hud/HudFileProxy.java#L139
We could just have:

https://zap/<random part>/display.<blah>.js
https://zap/<random part>/tools/attack.<blah>.js
https://zap/<random part>/images/hud.png

Would that work?

Nice! We could easily do something like prefix the hud files like https://zap/hud and https://zap/images. Or.. do they need to be in separate directories? The images path could be a child directory of the https://zap/hud.
If we could prefix all the files with a URL qualifier like the https://zap/hud and whitelist files in that path. All other files for any other ZAP plugins would still need to use the ?name= if needed?
We would actually still want this for inject.js actually because that's completely outside the Vue architecture. So that would still be https://zap/?name=inject.js unless decided otherwise, that's how it currently works. We only refactored Vue stuff but that's really just an iframe loader.

'images' can be a sub folder of the hud top level dir.
Are you ok with the top level random path element? If not we'll need core ZAP changes (and some more thinking;)

I think that could work. If I change the build option --public-url=https://zap/hud/RANDOM_TOKEN like this:

I get build paths like this, which the RANDOM_TOKEN could still be modified via string interpolation on first request when the HUD starts as it works now:

If that doesn't work out it looks like there's an option to customize the Parcel build routine to keep things the way they currently work but would take a bit of figuring out:
https://parceljs.org/api.html

Have you looked at quasar-framework, by the way? All this stuff is figured
out already.

@psiinon I'd like to pick this back up as it seems to be blocking a lot of other conversations/progress 😞  Has there been any progress of updating the API to allow for serving bundled assets?

@jaywon that would be great, we really need this :)
TBH I'd forgotten that you were waiting on API changes.
Am I right in thinking you needed the URLs to be changed to use path elements for the file names rather than parameters? I'll look at that asap.

Re https://github.com/zaproxy/zap-hud/pull/459#issuecomment-488229956 we will definitely need to set some variables at run time. We will also need to be able to dynamically add content, eg to support optional add-ons that want to add HUD functionality. Let me know if you need more details of these requirements.

Another thought - would it be possible to split this up into multiple PRs, eg closer to one of each of the issues you mention in the first comment?
Its ok if you cant, but it would make it much easier to review (and therefore merge) and that would probably cause less disruption to other PRs.

Let me rethink this. I've been watching PRs for front-end changes and know that there's commits that would need to get merged into this PR regardless. I think I started this PR w/ the intent of getting bundling and dependency management to work first, and it cascaded into the other refactoring, but now that the refactoring is clear maybe I can take a reverse approach to refactoring first w/ the end goal being the bundled assets and dependency management.
When #459 gets merged i can at least test that any of this is going to work and then map out a much clearer path than before. 🤔

@jaywon let me know if I can help out at all. Probably a lot here that's a one person job, but if even just sitting down and working through some small tasks and dropping inspirational quotes while you crank through the refactor helps lmk. Would be happy to get online at the same time and try to burn through some code with ya.

@jsoref Thank you for all the notes. It's unlikely that this PR will ever get merged (too large, outdated) as much as serve as a reference for a smaller, more up to date, series of commits. If I'm able to start that I'll take all of this feedback into consideration!

Hi @jaywon any plans to continue work on HUD?

@njmulsqb This is so old I'd have to refresh myself on it. I wouldn't mind but I believe the HUD project overall has stalled out?

@njmulsqb This is so old I'd have to refresh myself on it. I wouldn't mind but I believe the HUD project overall has stalled out?

Nop. HUD is looking for contributors but ZAP team lacks JS expertise, I have been trying to understand the codebase and contribute but since you've done so much work it will be a great help if you jump in.


Remove dependency on pipe events in HttpConnection

Refactored the HttpConnection to not depend on OnReaderCompleted and OnWriterCompleted. Instead we use ConnectionClosed to detect the FIN that we propagate via ConnectedAborted.
Fire ConnectionClosed when a FIN is received from both transports.
Remove pipe completion from Http1Connection and Http1OutputProducer. Instead just return from request processing.
Flow the CancellaitonToken property in RawStream

I'd also prefer to have this for https://github.com/aspnet/AspNetCore/pull/11109

@aspnet/build What's the deal with all the PRs, do we merge, do we wait? I know it's being looked at but what is the mitigation to unblock work?

@davidfowl as a few people have done, I suggest making a polite request for an admin to merge PRs where build success isn't being reported correctly or individual jobs hang. Looks like both of those issues have hit this PR but is it ready to merge?

@davidfowl as a few people have done, I suggest making a polite request for an admin to merge PRs where build success isn't being reported correctly or individual jobs hang. Looks like both of those issues have hit this PR but is it ready to merge?

I can merge I just want to know the protocol. Yes this is ready to merge.

:ok:, ping if you're blocked


Revert " this is the algorthm for radix sort in c++"

Requested the owner of the PR to move the file to appropriate location


Non exisiting directory in documentation of the simulation

In the documentation of the simulation setup, there is a path $FAB_ROOT/fabric_files/fabric_icarus_example/ given, but the directory fabric_icarus_example does not exists for me. In fabric_files I only have the directories FABulous_project_template_ for both verilog and VHDL and a generic directory.  Is the directory intended to exist or is the documentation of the simulation setup outdated?
However I have a Test directory in the project directory which seems to include all the test scripts, a test bench and a README.MD which to me indicates that the documentation is outdated.
Maybe @TaoBi22  can help, since you mentioned that you worked on the documentation before?

We have updated the testing structure to follow the "project" structure. The documentation will need to be updated.

Hey @IAmMarcelJung, sorry for the delayed response! This is indeed a holdover from the old structure of the project, I'll get it fixed. Thanks a lot for catching it!

Nice, thanks for fixing!


Scan against a .NET 5.0 Web Application encounters an unhandled literal type error

Interpreting results for csharp
Analysis produced the following diagnostic data:

| Severity |                          Message                           |
+----------+------------------------------------------------------------+
| error    | Unexpected C# extractor error: Unhandled literal type      |
| none     | (31 results for diagnostic "Successfully extracted files") |
Analysis produced the following metric data:

|               Metric                | Value |
+-------------------------------------+-------+
| Total lines of code in the database |  1345 |

/opt/hostedtoolcache/CodeQL/0.0.0-20210921/x64/codeql/codeql database print-baseline /home/runner/work/_temp/codeql_databases/csharp

The repo is public and can be accessed here.
The run in question is available here.
My workflow file was generated automatically by following the Setting up code scanning using actions section in the official documentation [here](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/setting-up-code-scanning-for-a-repository#setting-up-code-scanning-using-actions].
I encounter this issue whether I use automatic build or not, here is a workflow run and the workflow file for a workflow that uses a manual step and gets the same error.
Let me know if you need any further information.

Thanks for the detailed information! I believe you can safely ignore the Unhandled literal type error -- it is not critical enough to prevent the CodeQL database from being created and analysed.
However we would be interested in seeing which source code elements led to the error. Would you be able to modify your workflow to upload the SARIF results file, and share that with us? The change will look something like this:
- name: CodeQL analysis
  uses: github/codeql-action@v1/analyze
  with:
    output: results
- name: Upload
  uses: actions/upload-artifact@v2
  with:
    name: sarif
    path: results


PBS array job example?

Would it make more sense to control jobs using PBS array jobs?  For example, wouldn't the following be the easiest way to control a number of "rounds" of optimization with finite resources?
cat submit.sh
[...]
#PBS -l nodes=1:ppn=8
[...]
osprey worker tica.yaml -n 1 > osprey.$PBS_JOBID.log

qsub submit.sh -q gpu -t 0-25%3

I can file a PR if people think this is a clean way to do things...

Array jobs are only on SGE, right?

But yeah, I like them a lot. You can also use the -t obviously inside the job script, not just on the command line.

Our cluster is Torque / MOAB (https://github.com/cBio/cbio-cluster/wiki/MSKCC-cBio-Cluster-User-Guide)

Oh cool. I didn't realize torque had that feature too.


Some links appear to open in forkhub then immediately instead in chrome

Example: https://github.com/libass/libass/pull/236/commits/7d73cfea6590a0dbff29d837e8b621c8ed03335f
This happens when selecting this link from, say, atomic.
I don't recall seeing anything saying that forkhub can't view pull requests but it's entirely possible I've missed a part of documentation somewhere, not could I find anything in the open issues, though in part due to not knowing which key words were most relevant.
Please don't hesitate to ask for any further info if needed, or respectively discard the issue if it turns out to be known already. Thanks!

Well, after this link has been parsed and reformatted by forkhub here in the issues tracker, clicking it takes me to the expected page within forkhub. I suppose that means the issue must be somewhere else, but I've double checked that, e.g., forkhub definitely had all the default associations it should.
I'd appreciate and further advice on how to debug this, and in the meantime I'll try to see whether this behaviour depends on from which application I click the link...

ForkHub tries to open all http://github.com links, but when it can't open one (not supported or not recognized), it opens it in the browser.
The problem here is that ForkHub recognizes https://github.com/libass/libass/commit/7d73cfea6590a0dbff29d837e8b621c8ed03335f but doesn't recognize https://github.com/libass/libass/pull/236/commits/7d73cfea6590a0dbff29d837e8b621c8ed03335f.
It shouldn't be hard to fix.
For further reference #93


Thank you!

I can't wait to try this out. We've had nothing but misery trying to work with Allensby, with their expensive products, broken and user-hostile licensing system. I've used imagemagick and VIPS to create bitmap images but didn't realize this was the cause for the blurry/dithered CUPS labels.

Let me know how it works out for you! If it does work for you, I would also appreciate knowing the printer model you are using so that I can add it to the list.


is there a way to execute R script in the php file

I am trying to find the way connect R and php file. but i was unable. i used php exec() function also but it is not gives the output please help

Hi @msheranga! Could you please share the code you are trying to run or generate a small similar example?

thanks.
#test1.R
print("R programming")
#php file

but there is no output

@msheranga please have a look at the examples in this repo's README.md on how to use php-r. I don't see any mentioning of  Kachkaev\PHPR\... in your example, which means that you are not using the library. If you have any issue with php-r (not just combining PHP and R), feel free to further comment on this issue.


Upgrade to latest Rust (sort of)

Upgrading to Rust Alpha 2.
This PR purely fixes use of outdated syntax and typing semantics. Attempting to test spawns another truckload of compiler errors that seem more architectural in nature and warrant discussion before attempting to fix. Some compiler warnings about beta instability remain.
cargo build passes, but cargo test doesn't.
Squash available upon request. I won't merge this myself since it doesn't completely fix the build.

Awesome!

@cybergeek94 New nice profile picture! You are awesome.


failed to reserve filesystem for vm: failed to find a candidate to host vdisk of size

"failed to reserve filesystem for vm: failed to find a candidate to host vdisk of size '51200': Not enough space left in pools of this type ssd"
workload: https://explorer.devnet.grid.tf/api/v1/reservations/workloads/48557
network: devnet

Node didn't have enough capacity


Add URLs and descriptions for the COD and the TCOD

URLs and descriptions added for COD and TCOD.

Are the base URLs pointing to index meta-databases as required?

Not yet. I plan to roll out the index meta-database this week.

If you want this update to be available ASAP, you should probably point it towards master branch instead :)

As discussed in #214, I'm fine with develop :)

Not yet. I plan to roll out the index meta-database this week.

May I recommend you try out our new-fangled index meta-database implemention in opimade-python-tools? :smile:

May I recommend you try out our new-fangled index meta-database implemention in opimade-python-tools?

While I'm really interested in opimade-python-tools, I'm going to implement this myself ;) I think of this as yet another test for the specification :)

This should be closed here and remade in Materials-Consortia/providers.

Now that #222 is merged, this needs to be closed and refiled against Materials-Consortia/providers.

Now that #222 is merged, this needs to be closed and refiled against Materials-Consortia/providers.

Right, closing.


How to add channel administrator?

Hi there,
I can't find a method for adding channel administrators. Like that:

The method setChatMemberStatus can be used for most changes of the status.

The method setChatMemberStatus can be used for most changes of the status.

Thanks for your answer!
One more situation. How can I add bot to channel and make him administrator?

The method setChatMemberStatus can be used for most changes of the status.

The method setChatMemberStatus can be used for most changes of the status.

I'm trying to add the bot as admin with setChatMemberStatus, addChatMember, and addChatMembers. But every time I get the error: 400 USER_BOT.

You should use setChatMemberStatus and specify correct administrator rights. You are likely specifying empty list of rights in chatMemberStatusAdministrator and an administrator must have at least one administrator right.

You should use setChatMemberStatus and specify correct administrator rights. You are likely specifying empty list of rights in chatMemberStatusAdministrator and an administrator must have at least one administrator right.

OMG, you're right, thanks!


[BUG] Network Devices Renamed on Reboot and Link Failures with Intel E810-XXVDA2 Network Card on Supermicro SYS-740GP-TNRT

Describe the bug
After installing Harvester, the network devices in our system change their names after every reboot, which disrupts network bonding configurations. Additionally, some network devices remain down despite being physically connected. We believe the issue may be related to the Intel E810-XXVDA2 network card installed in our system.
This problem is preventing the node from joining the Harvester cluster. The rancher-system-agent logs show repeated warnings like the following:
Nov 18 16:38:46 harvester02-tamedai01 systemd[1]: Started Rancher System Agent.
Nov 18 16:38:46 harvester02-tamedai01 rancher-system-agent[7793]: time="2024-11-18T16:38:46Z" level=info msg="Rancher System Agent version v0.3.6 (41c07d0) is starting"
Nov 18 16:38:46 harvester02-tamedai01 rancher-system-agent[7793]: time="2024-11-18T16:38:46Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Nov 18 16:38:46 harvester02-tamedai01 rancher-system-agent[7793]: time="2024-11-18T16:38:46Z" level=info msg="Starting remote watch of plans"
Nov 18 16:38:46 harvester02-tamedai01 rancher-system-agent[7793]: time="2024-11-18T16:38:46Z" level=info msg="Starting /v1, Kind=Secret controller"
Nov 18 16:39:46 harvester02-tamedai01 rancher-system-agent[7793]: W1118 16:39:46.573537  7793 reflector.go:456] ... watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 17; INTERNAL_ERROR; received from peer") has prevented the request from succeeding

We have already attempted to resolve the issue by updating the network devices to the latest firmware, but this has not resolved the problem.
To Reproduce
Steps to reproduce the behavior:

Install Harvester on Supermicro Superserver SYS-740GP-TNRT hardware with Intel E810-XXVDA2 network card.
Reboot the server.
Observe changes in network device names and connectivity status.

Expected behavior
Network devices should retain consistent naming across reboots, all physically connected network devices should be active, and the node should successfully join the cluster.
Support bundle
Support bundle could not be generated, as the node is unable to join the cluster.
Environment

Harvester ISO version: v1.3.1 (https://github.com/harvester/harvester/releases/tag/v1.3.1)
Underlying Infrastructure: Baremetal, Supermicro Superserver SYS-740GP-TNRT with Intel E810-XXVDA2 network card

Additional context
The server has the following key hardware specifications:

2x Intel Xeon Silver 4309Y CPUs
2x Intel E810-XXVDA2
64 GB DDR4 ECC RAM
2x PNY NVIDIA RTX A6000 GPUs
Supermicro X12DPG-QT6 motherboard with redundant 2200W power supply

@Maxine-N
We had the same issue with same NIC, however not at every reboot
TIcket ref: https://github.com/harvester/harvester/issues/6808
We have yesterday done lab testings with RC5 and problem seems resolved for us with RC5 (or at least not longer reproducable). So should be fine with the official release of 1.4.0.
If you want, you can test RC5 to confirm as you seems to have the issue at every reboot.

Any idea when the 1.4.0 will be released?

Any idea when the 1.4.0 will be released?

Release: 11/27
So in a bit more as a week ;-)


Installation error [v1.9]

Hello,
I attempted to install Emmet but Brackets said me that an error has occured.
NOTE: I'm using an old version of Brackets because I'm on a school computer.
Brackets version: 1.9 édition 1.9.0-17312 (release 189f6d39a)

Hi @ghost!
I had the same issue yesterday. Please, make sure you disabled every other extension, removed the faild Emmet installation, and restarted Brackets before attempting to install Emmet again.
Worked for me.
Hope you'll find this helpful :)


Add support for disable scroll property

I would like to have some feature for removing scroll from component like scrollEnabled, or if exists some workaround for this solution.

This would make a lot of sense, given that scroll functionality is broken on Android.


enhanced proxy support => more flexible getRemoteServerRootPath()

Introduction
The newly introduced possibility of https://github.com/microsoft/vscode/pull/147311 and https://github.com/microsoft/vscode/pull/148707 allow VSCode to respect the full-path of its URL if it matches the pattern https//<myserver>/<quality>-<commit>/. One server can now serve multiple versions of VSCode - that is great!
Feature request
This would get an even broader use-case if the function getRemoteServerRootPath() could also allow to be less fixed to a specific pattern and instead just return the full url root-path as it is known from the current url.
Use-Case
We would like to run VSCode behind a configurable-http-proxy (JupyterHub) in combination with the jupyter-server-proxy + plugin which allows to proxy arbitrary local ports (like localhost:3000) to an extended URL of the running JupyterLab session.
With a logged in user a common URL to access VSCode/OpenVSCode running on localhost:3000 would for example look like this
https://jupyter-jsc-staging.fz-juelich.de/user/j.goebbert.fz-juelich.de/de47c0193b364c36b9da34e681f14911/openvscodeserver
which depends on the user-id and some hash and therefore changes with every new login & user.
Hence, in this example the function getRemoteServerRootPath() would need to return /user/j.goebbert.fz-juelich.de/de47c0193b364c36b9da34e681f14911/openvscodeserver to make the full setup work fine.

Links
To some extend this is linked to the issue https://github.com/gitpod-io/openvscode-server/issues/320

These are the errors from the current VSCode version:

The root path (yellow marked) is simply not respected (red underlines)

I'm also having this exact issue, and would like to run this behind a jupyterhub proxy.

Just briefly citing my related comment from https://github.com/gitpod-io/openvscode-server/issues/320 below.

I also stumbled across this problem... maybe one should mention that there is also a Visual Studio Code distribution code-server, which has patches applied (see also these search results) that make Visual Studio Code accessible on arbitrary base URLs, i.e. behind reverse proxies such as jupyter-server-proxy. I think, I will be using that one for now.

duplicate of https://github.com/microsoft/vscode/issues/165391


MinimumIdle affects MaximumPoolSize

AFAIU this is intended behavior that is present for years, but still create this issue so it can be closed as wontfix at least.
I have many microservices - some with many RPS and some with only a few RPM. They all use the same database with limited number of max connections. One day I faced a problem that some microservice could not start because all connections were exhasted on DB side. I used defaults at that time and MaximumPoolSize of 10 seemed reasonable default to me. But I overriden MinimumIdle to 1 from default 10 for all my microservices to meet DB requirements regarding max connections. And only on prod environment I discovered that this config also set MaximumPoolSize to 1 too.
Here is code sample
        try(HikariDataSource ds = new HikariDataSource()){
            ds.setJdbcUrl("jdbc:h2:mem:test_mem");
            ds.setMinimumIdle(1);
            try (Connection ignored = ds.getConnection()) {}
            System.out.println(ds.getMaximumPoolSize());
        }

And it prints 1.
Removing ds.setMinimumIdle(1); this code prints 10.
TLDR: please remove setting default MaximumPoolSize to MinimumIdle.

The relevant code is here. You just need to set the MaximumPoolSize as well and it will not be overridden.

@billoneil Yes, I just wanted to leave a note in issue-tracker that this may seem counterintuitive, while I do understand that it is done to promote fixed-sized pools.

Behaviour changed since HikariCP-3.3.1
https://github.com/brettwooldridge/HikariCP/commit/86612b4bc4fb84e9f2f417c22d11dcde84dd369e#diff-10c18d20c1364f48e0b2c8c7ae20f718
https://github.com/brettwooldridge/HikariCP/blob/dev/CHANGES

do not default maxPoolSize to minIdle when only minIdle is specified

@douph1 thanks, than this issue is resolved


@KtorDsl annotation

It would be nice annotate get, post, etc with @KtorDsl annotation.

Hi,  thanks for the hint. the annotation will be added in the next version.


Merge Dev 1.0.0 into Master

What is the purpose of the change

Update appconn
Update dss version to 1.1.0
Update linkis version to 1.1.1

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes)
Anything that affects deployment: (yes)
The Core framework, i.e., JobManager, Server.: (yes)

Documentation

Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? (docs)

LGTM.


Make Internet Explorer check compatible with uglify

The isOldInternetExplorer variable is always false if using uglify (and maybe other minification methods) since comments gets stripped. I suggest changing to another method for checking Internet Explorer versions.

sounds reasonable. PR welcome!

We ran into a similar issue here with the updated version of this library. Uglify was moving the @cc_on comments out from where they originally existed in the code, so it executes those checks outside of the initialization block. It's too late to assign variables, even if they did exist in the global namespace.
Since there's some user agent sniffing going on for iOS and Android, we just moved the checking for buggy and old IEs to the same format, specifically:
  var isBuggyIE = /MSIE [0-9]\./i.test(userAgent);
  var isOldIE = /MSIE [0-8]\./i.test(userAgent);

The regex is a bit hacky, but at least the idea should be sound, unless I'm missing something that you ran into. I guess you could do some more checking, like MS does here.
This might close #40 as well. If this sounds good, I'd be happy to do a PR.
Great library, by the way, saved us days of headaches.

looks good to me. any objections from @zoltan-dulac (who originally authored this part)? feel free to send a PR :)

thank you for the fix, it's released as v0.5.4

No problem, hope it works out.
--
Dillon Lucente
WILL Interactive
On July 15, 2015 at 12:31:44 PM, Rodney Rehm<EMAIL_ADDRESS>wrote:
thank you for the fix, it's released as v0.5.4
—
Reply to this email directly or view it on GitHub.

Great work, thanks! @dillonlucente @rodneyrehm


LLVM Cannot select error on PowerPC64LE

LLVM ERROR: Cannot select: 0x7fa027678410: ch = PPCISD::STBRX<ST6[%658](align=1)> 0x7fa02663ce38, 0x7fa0228a63a8, 0x7fa027678ea0, ValueType:ch:i48
  0x7fa0228a63a8: i32 = truncate 0x7fa027938820
    0x7fa027938820: i64 = srl 0x7fa027678750, Constant:i32<16>
      0x7fa027678750: i64 = or 0x7fa0277aebc8, 0x7fa026724680
        0x7fa0277aebc8: i64,ch = CopyFromReg 0x7fa029932440, Register:i64 %200
          0x7fa0279382d8: i64 = Register %200
        0x7fa026724680: i64 = shl nuw nsw 0x7fa027938410, Constant:i32<16>
          0x7fa027938410: i64 = AssertZext 0x7fa027938a28, ValueType:ch:i8
            0x7fa027938a28: i64,ch = CopyFromReg 0x7fa029932440, Register:i64 %202
              0x7fa027938c30: i64 = Register %202
          0x7fa027678958: i32 = Constant<16>
      0x7fa027678958: i32 = Constant<16>
  0x7fa027678ea0: i64,ch = CopyFromReg 0x7fa029932440, Register:i64 %158
    0x7fa0277b0068: i64 = Register %158
In function: _ZN6base646decode13decode_helper17h84289d425683d5b3E.llvm.15558209233772563096

https://travis-ci.org/LiveSplit/livesplit-core/jobs/375720556#L754
This was caused by Rust's upgrade to LLVM 6.
We probably want to turn off this CI Build in the meantime.

This is fixed in https://github.com/LiveSplit/livesplit-core/commit/a8093cfeed64caac48fd23d2dc34d432338b4a64


Feature request: support for server side rendered preview via ajax

Currently there is no callback function available to update the preview content once the ajax request to parse the original content has completed. Thanks!

Something like this:
        onPreview: function (e) {
          var content = e.getContent();
          $.post("/misc/preview",{data: content}, function (data) {
            e.setPreviewContent(data);
          });
        }

Hey @aruizca
The onPreview hook expecting string as return value, so i guess you could use synchronuous call to your server, and return the result at the end of the hook callback.

I've made a PR #306 that attempts to implement setPreview() as @aruizca has wanted.
Async call would be helpful so server-side rendering of markdown is identical to in-editor preview. Not all of us render and have javascript stack for whole applications, so this would be really useful.


Allow f32 and f64 map keys

Now we allow objects with f32 or f64 string-serialized keys. For example:
{ "1.123": "foo", "2.0": "bar" }

The test_deny_float_key test has been inverted, and the deserialize_integer_key macros have been renamed to deserialize_numeric_key in order to support f32 and f64.

Would you be able to elaborate on the use case for this?

This is useful for serializing data types like BTreeMap<NotNan<f32>, T>, using a type like NotNan or OrderedFloat from the ordered-float crate. A type like that could be useful in a few situations, e.g. a sorted map for timestamp -> event data, or to maintain an ordered list that allows for insertion between integer indices.


Fargate Platform 1.4.0 Support

Summary
Fargate platform 1.4.0 has been recently released and supports a new range of network related metrics. It would be great to see these included in the New Relic solution for Fargate container monitoring.
Desired Behaviour
This integration, when running on Fargate as a sidecar container, should send all metrics available on Fargate platform 1.4.0
Possible Solution
Add support for the Task Metadata Endpoint v4 for deployments running on Fargate platform 1.4.0
Additional context
We heavily use Fargate for our application deployments, and we use New Relic as our primary monitoring tool. However, it has been difficult to get detailed container-level metrics for things we care about. The v4 metadata endpoint for Fargate provides additional detail, such as network traffic statistics for containers. This would be really helpful to us.

It is quite confusing, but nri-ecs is in charge only to fetch metadata regarding tasks, the metrics are scraped by nri-docker in the aws fargate package.
I bumped the metadata version on that repo and opened a PR


Pre-bundle @alifd/next takes 5s

Describe the bug
I notice Vite 4 is slow when optimize a dependency @alifd/next. My own test result on a Macbook Pro:

Vite 4: vite optimize --force, 5.3s
Vite 3: vite optimize --force, 5.0s
Vite 2: vite optimize --force, 3.3s
esbuild: esbuild --bundle, 0.1s

It seems Vite's pre-bundling is getting slower in newer releases...
Reproduction
https://github.com/guoyunhe/test-vite-alifd-next
Steps to reproduce

Open http://localhost:5173/ in browser
Open browser's dev tools
npm i && npm run dev and check network traffic

System Info
System:
    OS: macOS 13.2.1
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Memory: 1.42 GB / 16.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 18.14.1 - /usr/local/bin/node
    Yarn: 1.22.17 - /usr/local/bin/yarn
    npm: 9.3.1 - /usr/local/bin/npm
  Browsers:
    Chrome: 111.0.5563.64
    Edge: 111.0.1661.44
    Firefox: 111.0
    Safari: 16.3
  npmPackages:
    @vitejs/plugin-react-swc: ^3.0.0 => 3.2.0
    vite: ^4.1.0 => 4.2.0

Used Package Manager
npm
Logs
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

Duplicate of https://github.com/vitejs/vite/issues/8850. Locally it's 2623ms for me, we've been working on this and the next minor should bring it to 224ms.


feat: Add fuzzy parameter in search query

What's the purpose of this pull request?
Adds the possibility to pass the fuzzy parameter in the search query
How to test it?
Change the parameter value between 0, 1 or auto and check the search result return

I used the carrefour account to test since that's where I found the problem, searching for the term 'Molico' was bringing up products like 'Molho de tomate'

Hey @PedroHenrique-git, thank you for your contribution. In your experience, would the value of fuzzy change for different pages or is it a store-wide configuration?

@gvc , I updated the pr

The IS team communicated us that the fuzzy parameter should not be set by the users of the API. We're also using it wrong internally and will fix this behavior in the future. If you want to always enable or always disable the fuzzy functionality, you should open a support ticket requesting them to do that on their side.
With this context this PR is no longer relevant.


Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index', 'flax_model.msgpack'] found in directory ./models/en_balanced_bertbase-g512-epochs3/ or from_tf and from_flax set to False.

Hello,
when i run the code , it report this error, i want to know if you offer this model?
and how can i explain my fine-turing model? can you off an example?
thank you so much!

How did you solve this problem ?


'beforeLeave' option - for cancel navigation by react lifecycle

v4
hi
I need 'beforeLeave' behavior to cancel navigation before the new component constructor will call.
I saw behavior of onEnter only -  by use componentWillUnmount as  you suggested in your documentation.
The use in componentWillUnmount is problematic for me because if I have for example server requests in the next componennt constructor they will call although I cancel it.
as you can see in the example (https://codesandbox.io/s/pp538kvnyx)) console - when I  cancel navigation in componentWillUnmount, the constructor of the new component is called and just then componentWillUnmount will call.

This is a bug tracker, not a support system. For usage questions, please use Stack Overflow or Reactiflux where there are a lot more people ready to help you out. Thanks!


Fix the rendering of the .cInlineImage tag

Description:
Need to change the MD format of all the images to HTML to avoid the .cInlineImage tag being rendered on the MD files as shown below.

E.g., <img src="./img/edit-the-code/notebooks/notebook-create.gif" class="cInlineImage-full"/>
Suggested Labels:

Suggested Assignees:

Affected Product Version:
OS, DB, other environment details and versions:
Steps to reproduce:
Related Issues:

Will be fixed via https://github.com/wso2/docs-ballerina-vscode-extension/pull/114.


Build nipkg of binaries

[X] This contribution adheres to CONTRIBUTING.md.

What does this Pull Request accomplish?
Enables building nipkg installers as part of the build pipeline.
Why should this Pull Request be merged?
We want to test the binaries that we would actually ship. This change enables us to test the actual installers that would be used in VeriStand systems.
What testing has been done?
Builds of branches master, release-18.1 and dev/package-build. Only master and release build nipkg installers and dev branches do not. Repos using the build tools still need to be updated with control files and new build.toml information to use the enhancements in this change.

@adchurch - This is only partial implementation of what you did for this pull request. If you have feedback about the package building part that I left out, please let me know. I do not have a manifest at the moment, but we want to get packages building to enable our work for automated tests.
We decided it was good enough for us to use the Jenkins build number because we will not be changing build servers, and if we do, we can begin builds on that server with the last build number, so we are not using a separate repo for configuration.

Hey @buckd, thanks for the FYI. I'm really excited to try this. We will use this in our GitHub org(s) if we can.
In my original PR I assumed that the .nipkg payload source directory would always be equal to the custom device build directory. This assumption doesn't always work if there are other files to include in the .nipkg besides the custom device. What we are doing now is using a map of paths in the build.toml file. This lets the user define an arbitrary number of (source_path, staging_path) pairs for the package payload.

@adchurch, I'll add multiple stage paths as a work item for us. The UDP custom device you referenced just uses a single staging path. Can you point me to a repo that requires multiple paths?


Just fix errors

There's no real reason to keep allowing errors if we can fix them on the fly. This will remove a bunch of error-reporting code, but that's fine. Less code is better.

Fixed in #37


[Java][FlightRPC] Handle binding parameters when server can't provide the expected type

Describe the enhancement requested
Based on apache/arrow-java#156, we need to support the case where the server can't specify the expected type of a given parameter.

Not all servers can always provide an accurate type for bind parameters. What should we do there? Note that ADBC uses NA/NullType as a wildcard/placeholder type here. This isn't specified in Flight SQL itself, but perhaps we could adopt that convention as well.

Proposal:

When the server doesn't know the type of any of the parameters, it can just set the parameter schema to null (or alternatively, an empty schema).
When the server doesn't know the type of only some of the parameters, it can just set the respective Field to NullType.
Before we start binding values, we transform the preparedStatement.getParameterSchema() based on the types of the given TypedValues. If the Schema is empty/null, we create every Field for the schema based on the TypedValue type. Otherwise, we replace all NullType fields with a Field based on the TypedValue.

The only potential drawback I see with this approach is that NullTypes can't be used for parameters. That being said, I can't really think of a case why one would want to do that.
Component(s)
Java

I've just run into this while adding various language integration tests against a FlightSQL server implementation. I'll note that so far both Go and Python haven't had any issues without a parameter schema available (though this did finally motivate me to implement that bit of the spec).

Before we start binding values, we transform the preparedStatement.getParameterSchema() based on the types of the given TypedValues. If the Schema is empty/null, we create every Field for the schema based on the TypedValue type. Otherwise, we replace all NullType fields with a Field based on the TypedValue.

I'm confused why the Java client is attempting to type check anything at all. I would think it should be up to the server to decide whether a provided type is acceptable or not. One obvious case of this would be whether type casting is allowed. Another case is when any type is acceptable like SELECT $1;.
I did spend some time reading through the implementation to see if I could provide a patch easily enough, but I'm no Java expert and got super lost in the whole arrow vector package trying to figure out how it's intended to work. I was vaguely surprised by the comment in VectorSchemaRoot acknowledging that most (all?) other languages use a RecordBatch for sending parameters. Given that its RecordBatch's that are sent across the wire I assume that's some sort of JVM optimization? No idea there.
Anywho, if anyone gets around to this, feel free to ping me as I should be able to fairly quickly check and proposed patches for fixing this.

IIRC, I believe there's a few things at play here:

The usage of the Avatica JDBC client
The statically typed nature of Java
The general design of the driver

I'd have to dig into the code again as it's been a while, but from what I remember, when you create a PreparedStatement, the driver creates a VectorSchemaRoot based on the returned Parameter schema. When you set a parameter, it attempts to set the given value on the VectorSchemaRoot. I'm not entirely sure why most of the Java implementation is around VectorSchemaRoot instead or raw ArrowRecordBatches, but I imagine it's due to some sort of memory optimization (being able to re-use allocated memory within the JVM for pointers to raw Arrow data in the shared memory space). That being said, I don't think the use of VectorSchemaRoot changes anything here as ultimately the data is indeed sent to the server in an ArrowRecordBatch.
I think it's possible to re-design how we bind parameters to do the following instead:

Move the creation of the VectorSchemaRoot from the AvaticaParameterBinder constructor into AvaticaParameterBinder.bind
Create the VectorSchemaRoot by introspecting the list of typedValues instead of using the PreparedStatement.getParameterSchema

The only potential issue here is I'm not entire sure if the FlightSqlClient needs the ParameterSchema to match the schema of the VectorSchemaRoot that we actually pass in.
I'm curious, how is this handled in other languages? Does the schema that is sent with parameters need to match the ParameterSchema?

Ok, I dug into this a bit more. I think the change shouldn't be too difficult, the hardest part will be agreeing on expected behavior.
I started a draft PR: #462
I would love to hear thoughts & opinions on:

Should we ignore the parameterSchema returned by the server altogether or should we respect all types except for Null vectors and treat those as "wild cards"?

What do the other drivers do?

How do we ensure all of the required Avatica TypedValues are mapped to an ArrowType?


tests: posix: common: separate posix threads base into a standalone test

Fixes #81491
posix.common contains testsuites that can be separated into smaller groups of tests. This change moves pthread, cond, key, mutex and mutex_attr into a singular testsuite at tests/posix/thread_base app directory.

This needs a minor change. I think it should only be merged after #81613


movetoworkspace previous doesn't do anything

hyprctl dispatch movetoworkspace previous
Does nothing when it should be moving the focused window to workspace previous
version:
Hyprland, built from branch main at commit 94b7b6b58475a0e802e390591f95e04c70aa676ddirty (added binds:focus_preferred_method). flags: (if any)

fixed in 4ea4efb871997a245539cae9d966ec8f07d3e907


RL integration loop

fixing issue #6
I used the rlglue library and implementation of tile coding agent from https://github.com/andnp/rl-control-template
looking for suggestions on how to further clean this test file

also, i could not figure out how to get rid of the unrelated commit from before, still trying to fix it :D

Looks like the build is breaking due to the new dependencies needed. Since these are dev-only dependencies, we can add those to the pyproject.toml here:
[project.optional-dependencies]
dev = [
    "pip",
    ...
    "PyExpUtils-andnp~=<whatever version>",
    ...
]

Unlike the other dev dependencies that we currently have, we'll probably want to version control these more tightly. The ~=4.0 pattern would lock the dependencies major version to prevent capturing breaking changes, while still letting us upgrade minor/patch versions.


Provide users with a less restrictive conda constraint

Description of Changes
I think that providing an equality constraint is dangerous for long term conda environments. Using ">=" should help alleviate that.

[ ] Wrote at least one-line docstrings (for any new functions)
[ ] Added unit test(s) covering the changes (if testable)

[ ] Included a screenshot or animation (if affecting the UI, see Licecap)

Issue(s) Resolved

Fixes #
Affirmation
By submitting this Pull Request or typing my (user)name below,
I affirm the Developer Certificate of Origin
with respect to all commits and content included in this PR,
and understand I am releasing the same under Spyder's MIT (Expat) license.

I certify the above statement is true and correct:

Screenshot of changes:


Required to check IS 6.0.0 compatibility

Is your suggestion a test requirement/change or improvements of the test framework? Please describe.

Ubuntun 2022 LTS and add ARM Architecture
Temurin 17
-MariaDB 10.4.13

Describe the solution you would prefer

Optional Fields
Additional context

Related Issues:

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Allow client to pass custom operationId

Thank you for submitting this pull request, however I do not see a valid CLA on file for you. Before we can merge this request please visit https://yahoocla.herokuapp.com/ and agree to the terms. Thanks! :smile:


How to upload header, body, params and files at the same time

Privileged issue

[X] I'm @tiangolo or he asked me directly to create an issue here.

Issue Content
here is my code.
If I only Use Request or UploadFile.It's OK. If I both use it. It raise 422.
server.py
import uvicorn
from fastapi import FastAPI, UploadFile, Request
from fastapi_restful import Resource, Api
from pydantic import BaseModel

class InputParams(BaseModel):
    request_uuid:str
    user_id:str = ""
    channelId:str
    SceneId:str

app = FastAPI()
api = Api(app)

class MyResource(Resource):
    async def post(self, request:Request, file: UploadFile):
        return "success"

api.add_resource(MyResource(), "/uploadfile")

if __name__ == "__main__":
    uvicorn.run(app)

client.py
import requests

files = {"file": ("server.py", open("server.py", "rb"))}
body = {
    "request_uuid": "test_uuid",
    "user_id": "test_user"
}
params = {"channelId":"test_channel", "sceneId": "test_scene"}
headers = {"Content-Type": "application/json;charset=UTF-8"}
response = requests.post("http://<IP_ADDRESS>:8000/uploadfile", headers=headers, json=body, params=params,files=files)
print(response.status_code)
print(response.json())

raise 422 Error
Due to the code specification, I can't write a lot of variables inside def
I ask chatgpt-4o。Then
server.py
import uvicorn
from fastapi import FastAPI, UploadFile, Request
from fastapi_restful import Resource, Api
from pydantic import BaseModel

class InputParams(BaseModel):
    request_uuid:str
    user_id:str = ""
    channelId:str
    SceneId:str

app = FastAPI()
api = Api(app)

class MyResource(Resource):
    async def post(self, request:Request):
        form = await request.form()
        file = form.get("file")
        print(type(file))
        # print(file.filename)
        return "success"

api.add_resource(MyResource(), "/uploadfile")

if __name__ == "__main__":
    uvicorn.run(app)

client.py no change.
and 200 success
But!!!
file is 'NoneType'
why？
And how I fix it?
thank you!!!

I solved.
Just don't pass the header.
But Why？

Oh!wrong way!
server.py
import uvicorn
from fastapi import FastAPI, UploadFile, Request, Form, File
from fastapi_restful import Resource, Api
from pydantic import BaseModel

class InputParams(BaseModel):
    request_uuid:str
    user_id:str = ""
    channelId:str
    SceneId:str

app = FastAPI()
api = Api(app)

class MyResource(Resource):
    async def post(self, file:UploadFile, request:Request):
        body = await request.json()
        print(file.filename)
        print(body)
        # print(file.filename)
        return "success"

api.add_resource(MyResource(), "/uploadfile")

if __name__ == "__main__":
    uvicorn.run(app)

client.py
import requests

files = {"file": ("test.py", open("test.py", "rb"))}
body = {
    "request_uuid": "test_uuid",
    "user_id": "test_user",
}
params = {"channelId":"test_channel", "sceneId": "test_scene"}
# headers = {"Content-Type": "application/json;charset=UTF-8"}
response = requests.post("http://<IP_ADDRESS>:8000/uploadfile", data=body, params=params, files=files)
print(response.status_code)
print(response.json())

error:
    raise RuntimeError("Stream consumed")
RuntimeError: Stream consumed

I can't fix it. Help!!!

I guess 'file' in 'Request'.But how I get it??


Channel pool release mechanics for producers

At the moment the channel pool is not actually a pool and instead is a mix of connection pool and channel factory. The channels should instead be releasable for short-term usage by producers in order to reduce the general channels count.

Releasing channels is not required since the Producers and Consumers are supposed to be long-lived. And even if they weren't, it is not a required resource utilization, it is not a big deal.


[BUG] Allow to translate "mute" word

Describe the bug
The "mute" word in the home screen popup filter menu is not translatable
To Reproduce
Steps to reproduce the behavior:

Go to main screen
Click on "All follows"
See "mute" word

Expected behavior
A way to translate this word in crowdin

I've also seen this. I believe it's the name of the list and not a word defined in amethyst.

Please @anthony-robin and  @davotoula do you still see this?

Please @anthony-robin and @davotoula do you still see this?

Hi @KotlinGeekDev, yes I still see it in the latest v0.78.2-FDROID version :)

I still see the "mute" word in my list. I don't believe it's a bug, it's the actual name of the list. You cannot translate that just like you cannot translate a hashtag?
A thing has changed though. I used to be able to see all muted threads when selecting this list.
It was a nice feature to view muted posts for anything interesting.

@davotoula I think a fix can be making the word a string resource, so that it can be translated.

But isn't the word coming from the relay actually?
We don't want different language clients have different names for that list?

Go for it! 😂
At the same time, can you look into why posts in that list are hidden now? They used to be visible!

@davotoula you're right. It seems to come from the relay.

Would it be possible to treat the mute list in a special way... And show a translated word in the client rather than just displaying what is returned from relay?
It could work if "mute" word is a standard in nostr... Otherwise it feels a bit unclean?

That's the thought I had as well.

In a sense, yes.

A bit like a Decorator.. Pass most list names through from the relay but capture "mute" and display it translated?

I actually found another way to do it: translate the word when it is being provided from the ViewModel.

The issue here is that many clients use a "d=mute" list from NIP-51's 30000 event kind, instead of the mute list from 10000, which has no name.
We can probably migrate to event kind 10000 moving forward and stop using the mute list. That will allow the app to have a mute entry and translate it via crowdin.

The issue here is that many clients use a "d=mute" list from NIP-51's 30000 event kind, instead of the mute list from 10000, which has no name.
We can probably migrate to event kind 10000 moving forward and stop using the mute list. That will allow the app to have a mute entry and translate it via crowdin.

I see. Makes sense.

Please @anthony-robin I dont know if this is still an issue for you. @vitorpamplona  maybe this can be closed already?(assuming clients have migrated to kind 10000 mute lists)

Let's close the issue. Thank you for working on this :)


Code clean up: use range-style for loops wherever possible

Remove variables obsoleted by commit 4f24617.

Thanks -- I started doing this but didn't get through it. I'll scrutinize this and most likely accept it.

There's a merge conflict you might want to fix after my latest push. Looks like maybe I had made one of the changes you also made in my recent work. At this moment, all my changes are in main, so it's a good time to squeeze this in. If you don't get to it, I'll take care of the merge conflict.

(I'm still doing active work -- I just pushed the most recent increment.)

Rebased to latest main.

Rebased to latest main.

Thanks! I should have time to merge it later today or sometime this weekend.


Make TW List component

Reset staging and merge 2186 into it, then the others that are in acceptance

The branch has been merged to main, but not staging because of different merging with scrollable background and the urgency to get out this fix to prod


For some notes, tasks are removed and not appended to the archive

I have "Archive to a separate file" set to true. When I select the archive tasks command, for many notes (maybe half of my notes) Obsidian writes an error msg "Error: file already exists" in the upper right corner, and completed tasks are just removed instead of being removed and then appended to the file with the corresponding task archive.
I've checked and the name of the note where the tasks to archive are located matches the name of the file with the task archive for the note. There's nothing special about the file with the task archive, it's writable, etc. Not sure why this happens.

This is happening on the latest version.

Hello, @alexlyzhov, thank you for your input! I need more information to reproduce the issue. Could you provide the following info:

The name of the file with completed tasks
The command you use
The name of the archive
What value you have in the archiver options here:

003 Resources/Scratchpad
Archiver: Archive tasks in this file
Other/Completed tasks/Task archive Scratchpad
Archive to separate file: on. Separate archive file name: Other/Completed Tasks/Task archive %.

I found the issue—it was because of the lowercase/uppercase mismatch in the archive file path (Completed tasks / Completed Tasks). After fixing the folder name I think the plugin started working.


Review and suggestions of the returned data (schema)

In this issue I propose to review the actual returned data by romcal, and suggest some improvements.
TL;DR: check the example under the "Proposed returned data" part.
It might be not too difficult to adjust the schema for existing features/data. And this could be a good opportunity to do it now with the codebase just upgraded to TS.
@pejulian and @tukusejssirs, let me know your thoughts.

Actual data :
Here the actual returned data (from the _F-TypeScript branch):
{
  "key": "fridayOfTheFirstWeekOfLent",
  "name": "Friday of the 1st week of Lent",
  "date": "2020-03-06T00:00:00.000Z",
  "type": 8,
  "data": {
    "season": {
      "key": "LENT",
      "value": "Lent"
    },
    "meta": {
      "titles": [],
      "psalterWeek": {
        "key": 4,
        "value": "Easter"
      },
      "liturgicalColor": {
        "key": "PURPLE",
        "value": "[object Object]"
      },
      "cycle": {
        "key": 0,
        "value": "Year A"
      }
    },
    "calendar": {
      "weeks": 53,
      "week": 10,
      "day": 65
    }
  },
  "_id": 834,
  "_stack": 0
}

A few comments on some properties:
→ type
Should be a string containing the key name of the rank of this celebration. Maybe it's also a good opportunity to rename it to rank (so it will be closer to the vocabulary used in the Universal Norms on the Liturgical Year and the General Roman Calendar).
→ _id and _stack
Ideally, they should not be output. They are used internally and should not be used outside of romcal.
→ data/season
Instead we might return an array:
[{ key: `LENT`, value: 'Lent' }]

Because some seasons can overlap. This is for example the case for Lent, the Easter Triduum and Estertide:

Lent end on the Holy Thursday just before the evening mass (In Cena Domini)
The Easter Triduum start from this mass, until Easter Sunday’s Vespers.
Eastertide starts the Sunday of Ester (included).

Official liturgical seasons are advent, christmastide, ordinary-time, lent, easter-triduum, eastertide.
In addition to seasons, we might consider offering another property for all kinds of periods. This periods property should also be an array, and might contain:

Official liturgical periods and octaves: christmas-octave, holy-week, easter-octave.
Other periods, for convenient usage: early-ordinary-time, later-ordinary-time.

→ data/calendar
In addition to the existing properties, some new property will be useful to get contextual data, especially for monastic usages:

dow (number) (or dayOfWeek) current day of week, 0 is Sunday, 6 is Saturday
'weekInSeason(number): the current week in a specific season. For example1forthursdayOfTheFirstWeekOfLent, 12forsaturdayOfTheTwelfthWeekOfOrdinaryTime, 0forashWednesday, 1forchristmas`...

Additionally, we could rename the existing properties to make them more explicit.

weeks to totalWeeksInYear (furthermore is it civil or liturgical year?)
week to weekInYear
day to dayInYear

→ data/meta/liturgicalColor
There is already a discussion about that here: #177
→ data/meta/psalterWeek
I'm wondering if key should start from 1 instead of 0, so 1 correspond to Week I. Also to be consistent with other metadata, should it return a string instead, for example WEEK_1? The value should be a localised string, actually this is not the case.
→ data/meta/cycle
Should be renamed to sundayCycle. Again here I'm wondering if the key should be a string containing something like YEAR_A. The value should be a localised string, actually this is not the case.
Additionally, we should consider to add a feriaCycle to get data one YEAR_1 / YEAR_2 (or YEAR_ODD / YEAR_EVEN).
→ data and data/meta properties
They are all data and metadata, so I'm wondering if everything inside meta could be at the same level of everything in the data object. Actually (at least for me), it's not very clear why some data are in data and some other are in meta. I suggest removing meta and move everything in data. And finally, for a reading perspective, I propose to rename data to meta (or metadata) to make this property name a little bit more different than the date property (I find the data and date names too similar).
The prioritized property has the same importance as the type property when romcal compute and organise celebrations for a specific day. This is in fact not a liturgical information, but a convenient trick to play with ranks and celebration precedences. So instead to be in the data object property, I suggest moving it to the root object (at the same level of type).
Proposed returned data:
{
  "key": "fridayOfTheFirstWeekOfLent",
  "name": "Friday of the 1st week of Lent",
  "date": "2020-03-06T00:00:00.000Z",
  "rank": "FERIA",
  "prioritized": false,
  "metadata": {
    "season": [
      {
        "key": "LENT",
        "value": "Lent"
      }
    ],
    "titles": [],
    "psalterWeek": {
      "key": "WEEK_1",
      "value": "Psalms week 1"
    },
    "liturgicalColor": {
      "key": "PURPLE",
      "value": "Purple"
    },
    "sundayCycle": {
      "key": "YEAR_A",
      "value": "Year A"
    },
    "feriaCycle": {
      "key": "YEAR_1",
      "value": "Year 1"
    },
    "calendar": {
      "totalWeeksInYear": 53,
      "weekInYear": 10,
      "dayInYear": 65,
      "dayOfWeek": 5,
      "weekInSeason": 1
    },
    "fromCalendar": "temporal",
  },
  "base": null // Or should not be output, because the current celebration is already a Feria
}

You will notice 2 others proposed properties:
→ metadata/fromCalendar
Represent the name from where this celebration is defined. E.g. "slovakia".
Or "temporal" when it comes from the "Seasons.ts" ; or "general" when it comes from "Dates.ts" or "calendars/general.ts".
→ metadata/fromCombinedCalendars
An array that contains the name of all the combined calendar. Actually, it could be one of them:
[ 'temporal' ]
[ 'temporal', 'general' ]
[ 'temporal', 'general', 'slovakia' ]

Then, when calendar inheritance support will be added (#48), this could look like:
[ 'temporal', 'general', 'france', 'paris', 'local' ]

The fromCalendar and fromCombinedCalendars properties will provide useful information to know from where a celebration is defined, and the final data for a day/celebration conputed from a combination  of calendars.
→ Other possible properties

metadata/temporal or metadata/sanctoral (or another generic name) that could include specific information from a future temporal/sanctoral db.
metadata/masses that contain readings (or at least references), prayers...

...
"type": 8,
...
"key": 4,
...

What are these keys, what do there mean? Could we not use some self-explanatory strings instead of the numbers?

@emagnier, I had the same thoughts after reading the JSON. :smiley: :+1:

dow (number) (or dayOfWeek) current day of week, 0 is Sunday, 6 is Saturday

I vote for dayOfWeek as it is more human-readable. :smiley:

weekInSeason (number): the current week in a specific season. For example 1 for thursdayOfTheFirstWeekOfLent, 12 for saturdayOfTheTwelfthWeekOfOrdinaryTime, 0 for ashWednesday, 1 for christmas...

And what about dayInSeason? :smiley:

weeks to totalWeeksInYear (furthermore is it civil or liturgical year?)
week to weekInYear
day to dayInYear

Because of the ambiguity in the year, I would suggest to following:

weeks to totalWeeksInCivilYear;
week to weekInCivilYear;
day to dayInCivilYear.

→ data/meta/psalterWeek
I'm wondering if key should start from 1 instead of 0, so 1 correspond to Week I. Also to be consistent with other metadata, should it return a string instead, for example WEEK_1? The value should be a localised string, actually this is not the case.

Why would you think it should be string? If the key is named psalterWeek, it is self-explanatory, and thus the Week (or WEEK) would be kind of a duplicity. I’d suggest to make it an integer value between 1..4.

→ data/meta/cycle
Should be renamed to sundayCycle. Again here I'm wondering if the key should be a string containing something like YEAR_A. The value should be a localised string, actually this is not the case.

I agree with the change of the key name to sundayCycle. But I would suggest to let the value be a single letter from [ABC]. I think that the localisation here is not necessary.

Additionally, we should consider to add a feriaCycle to get data one YEAR_1 / YEAR_2 (or YEAR_ODD / YEAR_EVEN).

I agree with the creation of it, but it should be named ferialCycle (note the letter l). And I suggest that the key value should be an integer (1..2) and no need for localisation then. KISS is my motto, brothers. :smiley:

→ data and data/meta properties
They are all data and metadata, so I'm wondering if everything inside meta could be at the same level of everything in the data object. Actually (at least for me), it's not very clear why some data are in data and some other are in meta. I suggest removing meta and move everything in data. And finally, for a reading perspective, I propose to rename data to meta (or metadata) to make this property name a little bit more different than the date property (I find the data and date names too similar).

I agree that the placement of the keys in either data or data/meta is quite ambiguous. On the other hand, I think that some division should be made:

celebration-specific data (like name, saint/blessed titles, liturgical colour);
season-related data (like season name, day in season, week in season);
breviary-related data (for now, only the psalter week);
Mass-related data (for the future usage);
liturgical-year-related data (Sunday and ferial cycles);
calendar-related data [like calendar name (e.g. cal1962), calendar ‘type’ (like general or particular), fromCalendar (like england or slovakia)].

"season": [
  {
    "key": "LENT",
    "value": "Lent"
  }
],

May I know why we have two keys (key and value) with essentially the same values (LENT and Lent)? I think it is a duplicity.
I think that season could contain both string and array values, that is: when there is only one season (which is usually the case), it should contain a string ("season": "Lent"); when there is more then one season, we should use an array:
"season": [
  "Ordinary Time",
  "Early Ordinary Time"
]

Same would apply to liturgicalColor, sundayCycle, ferialCycle.

Then, when calendar inheritance support will be added (#48), this could look like:
[ 'temporal', 'general', 'france', 'paris', 'local' ]

I like it very much. But I think we should add the continent too. Why? Because some of the calendar definitions should be made in the continent-specific celebrations (for the Patron of Europe and the like).

With the rest (stuff I have not commented on) I agree with @emagnier.

And what about dayInSeason?

I don't see a use case for that, but why not.

Why would you think it should be string? If the key is named psalterWeek, it is self-explanatory, and thus the Week (or WEEK) would be kind of a duplicity. I’d suggest to make it an integer value between 1..4.

It's already self-explanatory because it's nested in a psalterWeek object: psalterWeek.key and psalterWeek.value.

I agree with the change of the key name to sundayCycle. But I would suggest to let the value be a single letter from [ABC]. I think that the localisation here is not necessary.

Why isn't the localisation necessary? The localisation might not be necessary for everyone (it's why there is a key property). But also having a localised value will avoid to do translation specifically for that when we need it.

I agree with the creation of it, but it should be named ferialCycle (note the letter l).

I personally prefer feriaCycle: using feria as a noun. Here it's not a literary style but a coding style, where feria could be considered as a namespace, and cycle one of its properties.

May I know why we have two keys (key and value) with essentially the same values (LENT and Lent)? I think it is a duplicity.

Because key contain a constant which will never change. value is the localised equivalent. Here the example is in English so they are quite similar, but for example in French the value will contain Carême.

I think that season could contain both string and array values.

I will discourage that. One property should always have a unique type, not multiple possible types. This will avoid doing type checking when we consume data from romcal, and could also avoid potential bugs/issues.
Also we still need the { key, value } object. Because depending on our needs we could lean on a key containing a constant that will never change, or a localised value for display usage.

I like it very much. But I think we should add the continent too.

This is only an example. With the calendar inheritance architecture, we will be able to organise the calendar stacking as we want.

For other points, I will take time to think about it, and propose a revised data model (probably next Monday or Tuesday)

I was thinking about this issue while working on the separation of multiple celebrations.
It came to my mind that if we want to make the JSON response as agnostic as possible, should we not send all the metadata separately?
What I mean is that currently, we send the formatted celebration name as a string. I suggest that besides the formatted celebration name, we should send all the metadata the celebration name is composed of, also separately.
By the metadata I mean the following information (I suggest to create personMetadata[] array for persons and eventMetadata[] for celebration where necessary):

canonizationLevel (either saint or blessed);
numberOfPersons (how many persons are in the particular celebration; 0 for non-person celebrations);
names (if the celebration is a person celebration, we would output an array of names for the celebrated persons);
surnames (if the celebration is a person celebration, we would output an array of surnames for the celebrated persons);
sex (sex of the persons);
etc (for more metadata types, see this sheet).

Reason? If the user would have the separate metadata:

we would not have to provide a way to format the celebration name;
the user would be able to format the celebration name as they would want.

This is what I proposed to include under these properties:

metadata/temporal or metadata/sanctoral (or another generic name) that could include specific information from a future temporal/sanctoral db.

But I had not detailed them (because it's not something implemented yet)

Here an updated proposal of the returned schema.
Let me know your thoughts!
interface DateItem {
	key: string; // Uniq key name E.g. 'fridayOfTheFirstWeekOfLent'
	name: string; // Localised name. E.g. 'Friday of the 1st week of Lent'
	date: Date; // E.g. '2020-03-06T00:00:00.000Z'
	rank: 'SOLEMNITY' | 'SUNDAY' | 'TRIDUUM' | 'HOLY_WEEK' | 'FEAST' | 'MEMORIAL' | 'OPT_MEMORIAL' | 'COMMEMORATION' | 'FERIA';
	prioritized?: boolean; // Default: false (should be output only if true)
	base: null | DateItem; // Contain the base Feria or Sunday DateItem of the day. Is null if the current celebration is already a Feria or a Sunday

	metadata: {

		seasons: string[]; // [ 'EASTER_TRIDUUM', 'EASTERTIDE' ], or [ 'ORDINARY_TIME' ]
		seasonNames: string[]; // [ 'Easter Triduum', 'Eastertide' ], or [ 'Ordinary Time' ]

		cycles: {
			psalterWeek: 1 | 2 | 3 | 4;
			psalterWeekName: string;  // E.g. 'Psalter week 4'
			sundayCycle: 'A' | 'B' | 'C';
			sundayCycleName: string; // E.g. 'Year A'
			feriaCycle: 1 | 2;
			feriaCycleName: string; // E.g. 'Year II'
		}

		calendar: {
			totalWeeksInCivilYear: number;
			weekInCivilYear: number;
			weekInLiturgicalYear: number;
			weekInSeason: number;
			dayInCivilYear: number;
			dayInLiturgicalYear: number;
			dayInSeason: number;
			dayOfWeek: 0 | 1 | 2 | 3 | 4 | 5 | 6; // 0 is Sunday, 6 is Saturday
		}

		fromCalendar: string; // E.g. 'temporal', or 'general', or 'slovakia'..,
		fromCombinedCalendars: string[]; // E.g. [ 'temporal', 'general', 'france', 'paris', 'local' ]

		celebration: {
			liturgicalColor: 'WHITE' | 'GREEN' | 'RED' | 'PURPLE' | 'ROSE';
			liturgicalColorName: string; // Localised name. E.g. in English 'White', 'Green', 'Red'...
			titles: string[]; // E.g. [ 'DOCTOR_OF_THE_CHURCH' ]
			titlesNames: string[]; // E.g. [ 'Doctor of the Church' ]
			rankName: string; // E.g. 'Optional Memorial'

			// Except for liturgicalColor and titles, content below is TBD
			// the keys here might come from a future sanctoral and temporal db
			// and are given here as example
			type: 'temporal' | 'sanctoral';
			canonisationLevel?: 'Saint' | 'Blessed';
			firstName?: string;
			sex?: 'male' | 'female';
			number?: number;
		}

		masses: Mass[], // Array of mass, TBD (because some celebrations could have multiple masses, e.g. during Vigil)
	}
}

(I didn't want to divise data to much in nested objects)
Discussion:

metadata? or meta? or should we remove it and move all the content at the root of the DateItem object?
psalterWeekName, sundayCycleName, feriaCycleName might not be required. Do we want to keep them? On my side I vote yes.
feriaCycle or ferialCycle? On my side I vote for feriaCycle for the reasons I've gave above.
Anything else about property name, nesting?

I think that the values titles, celebration ranks and canonisation level should be in lowercase. Those words are not proper names (at least not in English).

1. `metadata`? or `meta`? or should we remove it and move all the content at the root of the `DateItem` object?

I don’t care if it’s metadata or meta, pick one. :smiley:

2. `psalterWeekName`, `sundayCycleName`, `feriaCycleName` might not be required. Do we want to keep them? On my side I vote yes.

We need to decide how much of the localisation we do in romcal. If we want to be minimalistic (in l10n and in the code), then drop them. If we want to have localised as much as possible, then keep them.
On my part, the basic localisation is enough, because (1) the more localisation, the the more flaws; (2) there should be external (client) app that would localise/format the data gathered/computed by romcal.

3. `feriaCycle` or `ferialCycle`? On my side I vote for `feriaCycle` for the reasons I've gave [above](https://github.com/romcal/romcal/issues/180#issuecomment-596004001).

I’d vote for ferialCycle (just because of English grammer). However, I am not opposed to use feria in the key name, but then I think it is better to change the order of the words: cycleSunday, cycleFeria.

4. Anything else about property name, nesting?

I am wondering if there is such a term as temporal in the liturgy. I know that we separate some of the celebrations are defined in the src/lib/Celebrations.ts instead of src/calendars/general.ts. I think the two files should be merged, because the celebrations in temporal/Celebrations.ts are part of the General Roman Calendar, even though they cannot be overriden. Even if the two files won’t be merged, it might be a good idea to set the value of fromCalendar and fromCombinedCalendars to either general or both general and temporal.

About your 4th point: in the liturgy their complete name is temporal cycle and sanctoral cycle.
The temporal (or the proper of seasons) is the essential and main part of the liturgical year (mainly sundays and ferial days). It's a cycle that traces the earthly life and the mystery of the Christ, and is deeply related to the liturgical seasons continuity.
They are also gathered in 2 books or 2 lectionaries: the lectionary of sundays (that also include solemnities that can occur on any week days); and the weekday lectionary.
In romcal, this cycle of celebrations is primarily computed in lib/Calendar.ts, and some of them in lib/Dates.js.
By opposition, the sanctoral (or the proper of saints), is an annual cycle of feasts, which consist of fixed dates (on the civil year), not related to seasons. It mainly contains the celebrations of the saints, and some other celebrations for the mystery of the church.
They are gathered in 1 lectionary: the sanctoral book.
In romcal, some celebrations from the sanctoral cycle are computed in lib/Dates.js (they are part of the general Roman calendar), and the majority are defined in src/calendars/* (including general.ts to complete the general Roman calendar, and many particular proper calendars).
Having a separated lib/Dates.js library gives us a handy manner to compute dates from each other (giving easy access to the main feasts and solemnity in the codebase).

FYI a new proposition has been made in the _F-TypeScript.
Please have a look to the corresponding documentation: https://github.com/romcal/romcal/blob/_F-TypeScript/docs/data-output.md#json-structure

Addressed on https://github.com/romcal/romcal/pull/250.


deadlock detected while deleting layer

PG::Error: ERROR: deadlock detected
DETAIL: Process 30804 waits for ShareLock on transaction 473226780; blocked by process 32674.
Process 32674 waits for ShareLock on transaction 473226783; blocked by process 30804.
HINT: See server log for query details.

+ 2 non-project frames
3
File "/home/ubuntu/www/production.cartodb.com/releases/20170202144204/config/initializers/zz_patch_reconnect.rb", line 51 in block in exec_delete
4
File "/home/ubuntu/www/production.cartodb.com/releases/20170202144204/config/initializers/zz_patch_reconnect.rb", line 62 in block (2 levels) in wrap_execute
+ 3 non-project frames
8
File "/home/ubuntu/www/production.cartodb.com/releases/20170202144204/config/initializers/zz_patch_reconnect.rb", line 61 in block in wrap_execute
9
File "/home/ubuntu/www/production.cartodb.com/releases/20170202144204/config/initializers/zz_patch_reconnect.rb", line 68 in with_auto_reconnect
10
File "/home/ubuntu/www/production.cartodb.com/releases/20170202144204/config/initializers/zz_patch_reconnect.rb", line 60 in wrap_execute
11
File "/home/ubuntu/www/production.cartodb.com/releases/20170202144204/config/initializers/zz_patch_reconnect.rb", line 50 in exec_delete
+ 17 non-project frames
29
File "/home/ubuntu/www/production.cartodb.com/releases/20170202144204/app/controllers/carto/api/layers_controller.rb", line 160 in destroy
30
File "/home/ubuntu/www/production.cartodb.com/releases/20170202144204/app/controllers/carto/api/layers_controller.rb", line 70 in map_destroy
+ 6 non-project frames
37
File "/home/ubuntu/www/production.cartodb.com/releases/20170202144204/app/controllers/application_controller.rb", line 100 in wrap_in_profiler
+ 73 non-project frames

https://rollbar.com/vizzuality/CartoDB/items/21610/?item_page=0&#instances

Closing automatically because of inactivity. If anybody considers that this still important, please reopen and it'll be prioritized.

@CartoDB/builder-backend it'd be great if next refactor step could address this bug.

Little research from last time this happened. It seems this is not entirely related with multiple ORMs, as we updated the layers controller to use only new models precisely for this but it kept happening.
Take a look at:
https://rollbar.com/vizzuality/CartoDB/items/21612/occurrences/21704486652/
https://rollbar.com/vizzuality/CartoDB/items/21445/occurrences/21704486584/
There are two simultaneous PUTs to the same layer, which is the problem. Maybe we should wrap the controller in a transaction, which should be possible now that everything uses the same model :)

Uhm, there's a loop with multiple DB operations without a transaction, that's not right. And you nailed it with those requests, they're blocking each other ✨
At first sight, a transaction should fix it. Nevertheless, if there are other operations that don't use that transaction it could lead to blocks within a single request, or fetching of old data, so this should be reviewed and tested with care.
Good catch!

Ok, finally found a likely explanation for the issue, but it's very hard to be sure. The problem here is not the lack of transaction. Although we should add them, this part of the controller (the loop) is only used for reordering layer, and is not the case in any of the deadlocks I found.
The main problem I get looking at Rollbar, is that you only get N-1 messages where N is the number of processes involved in the deadlock. That is because PostgreSQL kills all deadlocked queries except for one, that succeeds. The server log includes all N queries, but lacks context (ruby stacktrace).
Anyway, this is the case that I managed to reproduce, simultaneous requests to update and delete a layer. The update consists of adding/deleting an analysis node with a JOIN. What I see here is the user adding a JOIN with network problems, it doesn't work, so they delete the layer and retry.
What happens behind the scenes has to do with the layers and layers_user_tables relation. The update has register_table_dependencies in an after hook, while in the deletion, the dependencies must be removed before the layer (for FK consistency).
So what ends up happening is that the transactions run like this:

Update controller updates the layer
Delete controller removes layers_user_tables
Update controller tries to add/remove layers_user_tables. Starts waiting for a lock
Delete controller tries to remove the layer. Starts waiting for a lock. Deadlock is inmediately detected and this request is killed (500).
Update completes successfully.