Keras to estimator errors for RNNs

It seems like a bug that the following model results in the following assert:
The model (simplified from seq2seq):

import tensorflow as tf

params = {
    'seq_len': 32,
    'vocab_size': 50,
    'emb_act_size': 32,
    'rnn_act_size': 64,
    'out_size': 16
}

def mfunc(params):
    inputs = tf.keras.layers.Input(shape=(50, ), dtype='int32')
    emb = tf.keras.layers.Embedding(params["vocab_size"],
                                    params["emb_act_size"])(inputs)
    lstm1, s_h, s_c = tf.keras.layers.LSTM(
        params["rnn_act_size"],
        return_sequences=True,
        return_state=True,
        implementation=2)(emb)
    lstm2 = tf.keras.layers.LSTM(
        params["rnn_act_size"], return_sequences=True)(
            lstm1, initial_state=[s_h, s_c])
    y = tf.keras.layers.Dense(params["out_size"], activation='softmax')(lstm2)
    model = tf.keras.Model(inputs, y)
    model.compile(optimizer='sgd', loss='categorical_crossentropy')
    return model

x = mfunc(params)
tf.keras.estimator.model_to_estimator(keras_model=x)

The assert we are hitting in TF 1.10:
Traceback (most recent call last):
  File "Downloads/tmp.py", line 30, in <module>
    tf.keras.estimator.model_to_estimator(keras_model=x)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/keras.py", line 548, in model_to_estimator
    keras_weights)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/keras.py", line 449, in _save_first_checkpoint
    custom_objects)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/keras.py", line 315, in _clone_and_build_model
    model = models.clone_model(keras_model, input_tensors=input_tensors)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/models.py", line 263, in clone_model
    return _clone_functional_model(model, input_tensors=input_tensors)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/models.py", line 168, in _clone_functional_model
    **kwargs))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/layers/recurrent.py", line 526, in __call__
    self._num_constants)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/layers/recurrent.py", line 2590, in _standardize_args
    assert initial_state is None and constants is None
AssertionError

Here is the result of the environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
== cat /etc/issue ===============================================
Linux 97ddc7134519 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
VERSION="16.04.5 LTS (Xenial Xerus)"
VERSION_ID="16.04"
VERSION_CODENAME=xenial
== are we in docker =============================================
Yes
== compiler =====================================================
c++ (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
== uname -a =====================================================
Linux 97ddc7134519 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
== check pips ===================================================
numpy              1.14.5
protobuf           3.6.0
tensorflow         1.10.0
== check for virtualenv =========================================
False
== tensorflow import ============================================
tf.VERSION = 1.10.0
tf.GIT_VERSION = v1.10.0-0-g656e7a2b34
tf.COMPILER_VERSION = v1.10.0-0-g656e7a2b34
Sanity check: array([1], dtype=int32)

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce
Mobile device

I'm a colleague of @jmchen-g, and can update the issue while he is away:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):  Docker on Mac
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
TensorFlow installed from (source or binary): docker image tensorflow/tensorflow:1.10.0-py3
TensorFlow version (use command below):
Python version: python 3
Bazel version (if compiling from source): N/A
GCC/Compiler version (if compiling from source): N/A
CUDA/cuDNN version: N/A
GPU model and memory: N/A
Exact command to reproduce: See Below:

import tensorflow as tf

params = {
    'seq_len': 32,
    'vocab_size': 50,
    'emb_act_size': 32,
    'rnn_act_size': 64,
    'out_size': 16
}

def mfunc(params):
    inputs = tf.keras.layers.Input(shape=(50, ), dtype='int32')
    emb = tf.keras.layers.Embedding(params["vocab_size"],
                                    params["emb_act_size"])(inputs)
    lstm1, s_h, s_c = tf.keras.layers.LSTM(
        params["rnn_act_size"],
        return_sequences=True,
        return_state=True,
        implementation=2)(emb)
    lstm2 = tf.keras.layers.LSTM(
        params["rnn_act_size"], return_sequences=True)(
            lstm1, initial_state=[s_h, s_c])
    y = tf.keras.layers.Dense(params["out_size"], activation='softmax')(lstm2)
    model = tf.keras.Model(inputs, y)
    model.compile(optimizer='sgd', loss='categorical_crossentropy')
    return model

x = mfunc(params)
tf.keras.estimator.model_to_estimator(keras_model=x)

Running this yields the following trace:
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/keras.py", line 548, in model_to_estimator
    keras_weights)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/keras.py", line 449, in _save_first_checkpoint
    custom_objects)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/keras.py", line 315, in _clone_and_build_model
    model = models.clone_model(keras_model, input_tensors=input_tensors)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/models.py", line 263, in clone_model
    return _clone_functional_model(model, input_tensors=input_tensors)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/models.py", line 168, in _clone_functional_model
    **kwargs))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/layers/recurrent.py", line 526, in __call__
    self._num_constants)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/layers/recurrent.py", line 2590, in _standardize_args
    assert initial_state is None and constants is None
AssertionError

Here is some more system diag info from running tf_env_collect.sh
== cat /etc/issue ===============================================
Linux 97ddc7134519 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
VERSION="16.04.5 LTS (Xenial Xerus)"
VERSION_ID="16.04"
VERSION_CODENAME=xenial

== are we in docker =============================================
Yes

== compiler =====================================================
c++ (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

== uname -a =====================================================
Linux 97ddc7134519 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

== check pips ===================================================
numpy              1.14.5
protobuf           3.6.0
tensorflow         1.10.0

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.VERSION = 1.10.0
tf.GIT_VERSION = v1.10.0-0-g656e7a2b34
tf.COMPILER_VERSION = v1.10.0-0-g656e7a2b34
Sanity check: array([1], dtype=int32)

== env ==========================================================
LD_LIBRARY_PATH is unset
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
tf_env_collect.sh: line 105: nvidia-smi: command not found

== cuda libs  ===================================================

Thank you for providing such a clear minimal case.  cc @tanzhenyu @fchollet could you take a look?

Also @qlzh727 for recurrent cells.

And @k-w-w because this is happening in clone_and_build

Probably is a bug in the model_to_estimator.
Btw, the input of the model is not constructed correctly, namely the input shape.
The input should be in shape (batch, input_length), which should be (None, seq_len) in your case. The line tf.keras.layers.Input(shape=(50, ) will cause the emb to return (None, 50, 32) and the expected input to LSTM should be (None, 32, 50) "(batch, timestep, vocab)".

@k-w-w and @tanzhenyu, I did some debug and seems that when the lstm2 layer is loaded, the initial_state is showing up both in input as well as initial_state. Since the LSTM layer only accept one input tensor (batch, timestep, vocab), that's why it throws exception.
Do u have any insight about why the initial_state is picked up both as part of the input and also the kwarg ["initial_state"]?

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

Hi @k-w-w and @tanzhenyu Any update on this please? Thanks.

Adding @fchollet, as the error also appears when cloning the model using models.clone_model.

Nagging Assignees @fchollet, @tanzhenyu: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignees @fchollet, @tanzhenyu: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignees @fchollet, @tanzhenyu: It has been 44 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

I've run into the same issue, and it seem like  _clone_functional_model ends up passing the initial state as both inputs and as a keyword argument to the call/_standardize_args method of the Recurrent layer. (1)
A nasty workaround for us was to comment out the code extracting the initial state from the inputs as follows, since the shape of the state in the input list was not complete, but it was in the kwarg:
   def _standardize_args(...):   if isinstance(inputs, list):     # assert initial_state is None and constants is None     # if num_constants is not None:     #   constants = inputs[-num_constants:]     #   inputs = inputs[:-num_constants]     # if len(inputs) > 1:     #   initial_state = inputs[1:]     inputs = inputs[0]
...
(1) in the call to standardize_args when converting to an estimator I get this:
inputs: [<tf.Tensor 'time_distributed_7/transpose_1:0' shape=(1, ?, 320) dtype=float32>, <tf.Tensor 'input_state_h_2/input_state_h/Identity:0' shape=(?,) dtype=float32>, <tf.Tensor 'input_state_c_2/input_state_c/Identity:0' shape=(?,) dtype=float32>] initial_state kwarg: [<tf.Tensor 'input_state_h:0' shape=(1, 100) dtype=float32>, <tf.Tensor 'input_state_c:0' shape=(1, 100) dtype=float32>]

Nagging Assignees @fchollet, @tanzhenyu: It has been 15 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignees @fchollet, @tanzhenyu: It has been 30 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Can anyone help with this?

Can you call the model with some input and see it works before converting to the estimator? I suspect this is because initial_state is not passed to the first LSTM.

This is fixed with latest tf-nightly version '1.15.0-dev20190726'. Thanks!


remove pdf plugin name patch

Shouldn't be needed anymore because navigator.plugins returns a fixed list since Chrome 94. I tested tested using the demo page and got the same output on Google Chrome an Chromium.
navigator.plugins:
    PDF Viewer
    Chrome PDF Viewer
    Chromium PDF Viewer
    Microsoft Edge PDF Viewer
    WebKit built-in PDF

navigator.mimetypes:
    application/pdf
    text/pdf

https://chromestatus.com/feature/5741884322349056

@Ahrotahn @networkException @Zoraver Is this a significant enough change for us to wait for @Eloston 's approval?

I think the normal two approvals are fine since its just cleanup


Freedos package

Any chance of a freedo package ?
http://fdnpkg.sourceforge.net/
Just realised the license is bsd, so they would probably have one if it existed

I have no idea how I would do that. who maintains freedos packages?

It's basically a zip file with the format mentioned here
http://wiki.freedos.org/wiki/index.php/Package
I think usually the authors maintain the packages, I guess once it works it may be worth announcing it on one of the freedos mailing lists.

With some info from Matseuz Viste I've started experimenting with this + made a bash script that creates a package ANIMATOR.ZIP -
https://github.com/stuaxo/animatorpro-fdpkg
fdpkg has the idea of shortcuts to executables, since V CROP and PLAY might be the same as many other things, I've mapped them as ANIMATOR ANIMCROP ANIMPLAY.

cool. i'll have to work out how to run freedos to test it out

To test it in any DOS you just need fdinst.exe from fdpkg and set DOSDIR to point at a real directory somewhere.
There's a copy of ANIMATOR.zip here https://www.dropbox.com/sh/u85evagxfjz7dv5/AADbIUConZFD1ZtzGBpSabtUa?dl=0
Which I generated using the scripts I linked above.

The packaged version looks like the original? What about a PRO package? Maybe it makes more sense to only provide a package for the PRO version, that I assume is better in every way? Or have two packages or include both in the same package maybe?

Oh wow, good sleuthing - and thanks past-me for grabbing that.
I think (may be wrong) - but all these names, are just the same thing, with AnimatorPro being the current name, Auto desk animator, from when it was licensed to them, and V from before (see the different names in the README https://github.com/AnimatorPro/Animator-Pro).
It might be worth seeing how well this works, I seem to recall issues with it - especially around the shortcuts that fdpkg installed that are supposed to launch bits, but maybe it''s fine.
It maybe worth checking out The fork @kattkieru has too - though I don't know if that still runs in DOS or is only for modern systems
https://github.com/kattkieru/Animator-Pro

Is @kattkieru's fork fully functional? I'd love to know more about it, but it seems there isn't a proper readme for it.

The Cmake file looks like it only has been built for IOS and MacOS.   In theory you should be able to build it by checking it out then in the project folder and running:
$ mkdir build
$ cd build
$ cmake ..
$ make

I'm not sure what you end up with.

Guess if issue #27 is solved then it will be easy to make a FreeDOS package?
@stuaxo did you manage to build using OpenWatcom?

I have not tried that, though I have - at some point in the last year got OpenWatcom setup[1], so maybe I should try it.
[1]
I ported some of setup-openwatcom to bash:
https://gist.github.com/stuaxo/c140314d7ea07b32865df354a67ef188
And you can use it to install openwatcom in linux (only 1.9), as another way of installing it (though there are definitely more comprehensive ways of doing this).

I wouldn't bother with the main branch of my fork; all the work is currently in feature/poco.  The TLDR is I got an M3 Max laptop, and the entire program breaks on Apple Silicon, so that put a damper on my plans and it's taken some time to get to a point where things aren't crashing.
Current status is that it builds and runs fine on macOS, but a lot doesn't work: Loading and saving files is broken, and many menu items still crash the app.  Poco is a mess.  I have most of the VM working (apart from variadic functions), but I have to integrate it into the main app now; it's not at all properly connected.  Also haven't sorted how to remove the Poco menu when building -DWITH_POCO=off...
It will probably build on modern Windows? I haven't tried.  It's on the list after proper undo and the iPad port, which are the only things that I personally care about.  I'm deliberately using portable libraries to make this work, and everything is vendored in, so you don't need to download any extra dependencies.
You are free of course to use my CMakeLists.txt to bootstrap an OpenWatcom build, although I have no idea how that would work.  I have zero interest in building for DOS and don't really understand why anyone else wants to.  That said, the pristine pro folders are what you'd want, not the stuff I have in my fork because I'm slowly removing code that makes DOS / Amiga assumptions and replacing it with more portable stuff (see: how Poco called compiled functions with assembly; god-tier hack for the 90's).
As for my fork, I'll get around to Windows at some point but as nothing is super usable now, if you want to use the app your best bet is just using the zip of the original version with FreeDOS and not worrying about any new additions.

Someone is helping me prepare Windows and Linux versions of the MacOS port. There'll be a PR for that soon.
In the meantime, is there a compiled build of PJ that I can use? I'm not very experienced with DOS so I couldn't figure out building yesterday.

Someone is helping me bring the MacOS fork to Windows and Linux. There'll be a PR for that sometime in the coming days.
In the meantime, is there a compiled version of PJ that I can use? I'm not very experienced with DOS so I was unable to get it building myself yesterday.

That actually makes a lot of sense.
On the other side of it, for me, I’m aware that HD and some of the things I want to do in Animator Pro in the long run will require hardware acceleration to make useful. Also, the porting work I’m doing would also be a good base for Android if someone (not me) wanted to put in that effort at some point.
Anyway, I wish you luck. Pristine Pro should still be buildable with Watcom; it’s just a matter of sorting out their build scripts.
As for patches, @dreamyshorpy if you meant against my fork I’d prefer to do the windows port myself and don’t have an environment to test Linux. Please feel free for for my fork, though.

I'd again like to ask if anyone has gotten PJ (Animator Pro) to build, even if it's with proprietary Watcom.

everything in this thread gets forwarded to my email. I am sorry that i have been a little absent from this project. At the point of @Kattkieru’s fork split off, wangds had done a lot work to port the assembler files to c, thus reducing the project’s dependence on a very difficult to obtain assembler  called Pharlap with its own weird dialect; designed to work with its particular dos extender and REX system (relocatable executable- a pre windows DLL -like system)
after porting, wangds wrote presumably compilation instructions in a readme that i never had the time to test. if you can find that state of the repo, you can test it. I need to set aside some time myself to do work i have always intended to do: snow that animator pro is set up as an organisation I’d like to split this repo out into multiple repos for different things, so conversations like this can be a little less confusing and muddled

everything in this thread gets forwarded to my email

I understand. It's just that the thing I want to ask about isn't completely related to Animator Pro, so I didn't want to clog up the issue with unrelated chatter, but essentially I'm looking for someone with knowledge of the Animator Pro codebase who can help me create a fork of it for a single bespoke purpose. I don't want to take this issue further off topic, so if anyone's interested in hearing more details, please email me at [my username] at proton dot me.
-------- Original Message --------
On 5/14/24 10:08 PM, BriSeven  wrote:

everything in this thread gets forwarded to my email. I am sorry that i have been a little absent from this project. At the point of @.***(https://github.com/kattkieru)’s fork split off, wangds had done a lot work to port the assembler files to c, thus reducing the project’s dependence on a very difficult to obtain assembler called Pharlap with its own weird dialect; designed to work with its particular dos extender and REX system (relocatable executable- a pre windows DLL -like system)
after porting, wangds wrote presumably compilation instructions in a readme that i never had the time to test. if you can find that state of the repo, you can test it. I need to set aside some time myself to do work i have always intended to do: snow that animator pro is set up as an organisation I’d like to split this repo out into multiple repos for different things, so conversations like this can be a little less confusing and muddled
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

Real life keeps interfering and then I forget there was a conversation on here until I scroll down enough on my GH.
I have built AnimatorPro and managed to run it, a long time ago - but can't say I know the codebase.

@BriSeven Do you mean that wangds managed DOS compiles without assembly, with working poco and rex?  That would be amazing.
If folks really want to go down the DOS road, I'd recommend commenting rex and poco out to start.  My feature/poco branch has a WITH_POCO define that shows (I think) all of the spots.  That will simplify things and should make the app work with other DOS extenders.  I'd recommend attempting with Watcom 10.6 / dos4gw or djgpp / causeway.  If rex really, really, really is needed (and I'd question that), Pharlap is on the Internet Archive.
That said, I still think the best path forward for everyone is to use more portable code and compile for target machines.  The feature/poco branch uses libffi to get around the need for esoteric assembly, allowing the project to target almost any compiler / os / arch combo out there.  I have the poco interpreter working; it's just functions with variadic calls that are an issue at the moment (stuff like printf(const char* msg, ...) <-- the ... bit is causing me guff).
Also, with portable code rex problems go away.  We could use dlopen / LoadLibrary to directly open modern shared libs on all platforms.  The way both poco and libffi work would help with this.
@dreamyshorpy If you want to configure the app for a single use, your best bet (currently) is to make a Poco script and ship that script in a bundle with the original DOS build.  The UI is kind of hard coded, and until it's replaced with more flexible code, reconfiguring it will be a pain.
In terms of community, @BriSeven how would you feel about starting a Discord?

@kattkieru Could I theoretically use pocoscript as a mouse macro?
Also, I'm all for the creation of a Discord.
-------- Original Message --------
On 5/15/24 8:54 AM, Charles Wardlaw  wrote:

@.***(https://github.com/BriSeven) Do you mean that wangds managed DOS compiles without assembly, with working poco and rex? That would be amazing.
If folks really want to go down the DOS road, I'd recommend commenting rex and poco out to start. My feature/poco branch has a WITH_POCO define that shows (I think) all of the spots. That will simplify things and should make the app work with other DOS extenders. I'd recommend attempting with Watcom 10.6 / dos4gw or djgpp / causeway. If rex really, really, really is needed (and I'd question that), Pharlap is on the Internet Archive.
That said, I still think the best path forward for everyone is to use more portable code and compile for target machines. The feature/poco branch uses libffi to get around the need for esoteric assembly, allowing the project to target almost any compiler / os / arch combo out there. I have the poco interpreter working; it's just functions with variadic calls that are an issue at the moment (stuff like printf(const char* msg, ...) <-- the ... bit is causing me guff).
Also, with portable code rex problems go away. We could use dlopen / LoadLibrary to directly open modern shared libs on all platforms. The way both poco and libffi work would help with this.
@.***(https://github.com/dreamyshorpy) If you want to configure the app for a single use, your best bet (currently) is to make a Poco script and ship that script in a bundle with the original DOS build. The UI is kind of hard coded, and until it's replaced with more flexible code, reconfiguring it will be a pain.
In terms of community, @.***(https://github.com/BriSeven) how would you feel about starting a Discord?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

I've went ahead and made an unofficial one for the time being. https://discord.gg/jPemwfXF If anyone in charge wants me to transfer ownership so that it can be official, I'll happily oblige.

Has there been any updates since mid may? Can it be built on GNU/Linux (Such as debian) or in a windows 2000 virtual machine? The build scripts are completely scattered, so it's a pain right now to figure the pieces out. I know scrolling up someone built v, but not the pro version which has more functionality.

@stuaxo
@kattkieru
I'm more than willing to use a DOS environment like DOSBox, but I like to also use a touchscreen tablet. It is very spotty with getting it treated as no different than a regular mouse, like for instance, the cursor is fine with my USB mouse, but it jumps around with the tablet. Is there a remedy for this?

What's rather odd, @kattkieru , is that with other programs  like deluxe paint 2, it works fine. And this touchscreen tablet is not a wacom tablet, it's a huion device. But like I said, it works fine with deluxe paint 2, not sure why it's weird in Animator Pro.

Windows/macOS/Linux/Android support is unfortunately not of much help for building a FreeDOS package (see ticket title).
Maybe someone could get the GitHub compilation action working for that, as suggested by @stuaxo in issue #27? Or maybe the best approach is to ping the FreeDOS people instead and maybe someone there that has experience building old software for FreeDOS can figure this out? I do not remember if anyone from that project has commented on including Animator yet?


Fix dynamic vm transfer transaction

Description

Fixes transaction failure case when moving Flow from EVM to Cadence with primary balance coming from EVM by leaving enough FLOW in Cadence to cover storage consumption.

For contributor use:

[x] Targeted PR against main branch
[x] Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
[x] Code follows the standards mentioned here.
[x] Updated relevant documentation
[x] Re-reviewed Files changed in the Github PR explorer
[x] Added appropriate labels

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 86.73%. Comparing base (02aab3f) to head (b8f3e08).
Report is 183 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #138      +/-   ##
==========================================
+ Coverage   86.19%   86.73%   +0.53%
==========================================
  Files          18       18
  Lines         884      965      +81
==========================================
+ Hits          762      837      +75
- Misses        122      128       +6

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Fix gql server unnecessary nullable variables

When read from db, the arrays shouldn't be nullable unless specified

Also check orm relationships


DNS Challenge not working - Seems to get wrong DNS nameservers

Hello,
It hangs at:
Monitoring _acme-challenge.name.subdomain.domain.com. DNS propagation: ns-cloud-a1.googledomains.com.:53 ns-cloud-a2.googledomains.com.:53 ns-cloud-a3.googledomains.com.:53 ns-cloud-a4.googledomains.com.:53

Where subdomain has it's one zone and domain is also a zone, both at google cloud.
While it is monitoring it goes for ns-cloud-a?.googledomains.com but that is the nameservers for the domain.com
The name servers for subdomain.domain.com are ns-cloud-c?.googledomains.com
And when it times out, it changes the challenge so it hangs again and the process never completes.
I've tried also with a different subdomain and it gets the domain name servers for it, so the same error occur.
Is there any thing that can be done?
Thanks

It seems this commit:<PHONE_NUMBER>4a4abdca88bb755daebf03180f6c9d
Makes it always goes for the TLD, but it shouldn't be the case when a sub domain is hosted at different NameServers.


textarea autoresize when changing content dynamically

As per http://materializecss.com/forms.html#textarea , you need to trigger the "autoresize" event manually when updating textareas programmatically.

@awexfwex thanks for the note. A PR would be wonderful, if you're interested in contributing - more than happy to provide guidance on how to do that.
Cheers.

@awexfwex thanks heaps for fixing this.
Are you able to confirm this has been fixed in the latest released version and close the issue?

No, the previous pull request wasn't properly detecting changes for all scenarios. It was working for my specific scenario, but #172 addresses this.
Are there any automated tests?

There aren't any at the moment. I had a few a while ago, but after having some trouble with TravisCI I omitted them. We do need some :)
Merging now, so you can test again.

I've given it some smoke tests, but nothing thorough. Seems fine.


Soloseng/update-renovate-base-branch

Description
Primitive has transitioned from using master as the base branch, to using release/core-contract/<VERSION>
This PR updates the Renovate configs to reflect this change and only use the latest release branch.
One concern I have, and couldn't find a reference in the docs, is wether or not there will be a conflict when master and release/core-contract/<VERSION> have different configs.

closed in favor of #11044


[Question]  Hello, is there any size limit for sending pictures or files? I didn't find relevant information in the documentation

And I found that webjs supports more formats when sending videos, while noweb can only send MP4 format.

Hi!

is there any size limit for sending pictures or files?

The same limits that WA has, but if something doesn't work - feel free to open the issue!
I remember some limits in HD videos or big files exists in the WA apps itself :thinking:

noweb can only send MP4 format.

Yea, WEBJS (browser) converts it to mp4 internally, while NOWEB waits it in the right format.
https://waha.devlike.pro/docs/how-to/send-messages/#send-video-imagesversionspluspng
ffmpeg -i input_video.mp4 -c:v libx264 output_video.mp4

Probably we could convert the files as well, feel free to open the feature request and let's see how many people are intersted in it!


Error: P1002 database server was reached but timed out

Bug description
I was running prisma migrate dev and ran into this error.
Error: P1002

The database server at `xx.xx.xx.xx`:`5432` was reached but timed out.

Please try again.

Please make sure your database server is running at `xx.xx.xx.xx`:`5432`.

I'm not using pgBouncer. This is my connection string:
postgres://<XXXXXXXXXX>/defaultdb?connection_limit=3&connection_timeout=30&pool_timeout=30
I looked at #6053

How to reproduce
This is a random error.

Expected behavior
Primsa migrate happens as expected

Prisma information

Environment & setup

OS:  Mac OS 10.15.7
Database:  Postgres
Node.js version:  15.14.0
Prisma version:

prisma               : 2.21.2
@prisma/client       : 2.21.2
Current platform     : darwin
Query Engine         : query-engine e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/@prisma/engines/query-engine-darwin)
Migration Engine     : migration-engine-cli e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/@prisma/engines/migration-engine-darwin)
Introspection Engine : introspection-core e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/@prisma/engines/introspection-engine-darwin)
Format Binary        : prisma-fmt e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/@prisma/engines/prisma-fmt-darwin)
Default Engines Hash : e421996c87d5f3c8f7eeadd502d4ad402c89464d
Studio               : 0.371.0

Can you do other things (Introspection, Queries) with this connection string?

Interesting.
For introspection, I'm getting P1001 error
Queries and Mutations work fine though.

I deleted node_modules folder and reinstalled the packages.
Then I ran prisma migrate, got this error
Error: P1017

Server has closed the connection.

I just ran the command again, it's worked.

This all points to your database server being only reachable and stable sporadically. Could that be possible?

I don't think so, cause other queries and mutations work at the same time.

Can you summarize what works, and what does not?
All of these are using the same connection string setup?

I'm using the same connection string for development and migrations.
Queries, mutations are working all the time.
Migrations are throwing the above error randomly.

@s-kris Can you please tell us where your database is hosted? As you can guess it would be quite hard to reproduce this without knowing all the details.
Also, run export DEBUG=* before running migrate and paste the output here.

Hi @pantharshit00
My dev-database is hosted on hetzner cloud.
I'll post the debug log when the error is thrown (it seems be random as of now)

Hi @pantharshit00 I am running into the same issue and it is happening consistently. Issue only started as of the last 5 minutes. I rolled back all changes in that timespan but the issue still persists
My DB is hosted on Supabase
Debug log below:
prisma:loadEnv project root found at /Users/ceb/projects/kana-api/package.json +0ms
  prisma:tryLoadEnv Environment variables loaded from /Users/ceb/projects/kana-api/.env +0ms
Environment variables loaded from .env
  prisma:engines using NAPI: false +0ms
  prisma:engines binaries to download query-engine, migration-engine, introspection-engine, prisma-fmt +0ms
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "postgres", schema "public" at "db.xxxxxxx.supabase.co:5432"

  prisma:migrateEngine:rpc starting migration engine with binary: /Users/ceb/projects/kana-api/node_modules/@prisma/engines/migration-engine-darwin +0ms
  prisma:migrateEngine:rpc SENDING RPC CALL {"id":1,"jsonrpc":"2.0","method":"devDiagnostic","params":{"migrationsDirectoryPath":"/Users/ceb/projects/kana-api/prisma/migrations"}} +5ms
  prisma:migrateEngine:stderr May 03 17:08:57.119  INFO migration_engine: Starting migration engine RPC server git_hash="e421996c87d5f3c8f7eeadd502d4ad402c89464d" +0ms
  prisma:migrateEngine:stderr May 03 17:08:57.503  INFO quaint::single: Starting a postgresql connection. +384ms
  prisma:migrateEngine:stderr May 03 17:08:58.375  INFO DevDiagnostic:calculate_drift:sql_schema_from_migration_history: quaint::single: Starting a postgresql connection. +871ms
  prisma:migrateEngine:stderr May 03 17:09:03.093  INFO DevDiagnostic:validate_migrations:sql_schema_from_migration_history: quaint::single: Starting a postgresql connection. +5s
  prisma:migrate:dev {
  prisma:migrate:dev   devDiagnostic: '{\n  "action": {\n    "tag": "createMigration"\n  }\n}'
  prisma:migrate:dev } +0ms
  prisma:migrateEngine:rpc SENDING RPC CALL {"id":2,"jsonrpc":"2.0","method":"applyMigrations","params":{"migrationsDirectoryPath":"/Users/ceb/projects/kana-api/prisma/migrations"}} +9s
  prisma:migrateEngine:rpc {
  prisma:migrateEngine:rpc   jsonrpc: '2.0',
  prisma:migrateEngine:rpc   error: {
  prisma:migrateEngine:rpc     code: 4466,
  prisma:migrateEngine:rpc     message: 'An error happened. Check the data field for details.',
  prisma:migrateEngine:rpc     data: {
  prisma:migrateEngine:rpc       is_panic: false,
  prisma:migrateEngine:rpc       message: 'The database server at `db.xxxxxxx.supabase.co`:`5432` was reached but timed out.\n' +
  prisma:migrateEngine:rpc         '\n' +
  prisma:migrateEngine:rpc         'Please try again.\n' +
  prisma:migrateEngine:rpc         '\n' +
  prisma:migrateEngine:rpc         'Please make sure your database server is running at `db.mqwxclxnqubbhcssyjnf.supabase.co`:`5432`.\n',
  prisma:migrateEngine:rpc       meta: [Object],
  prisma:migrateEngine:rpc       error_code: 'P1002'
  prisma:migrateEngine:rpc     }
  prisma:migrateEngine:rpc   },
  prisma:migrateEngine:rpc   id: 2
  prisma:migrateEngine:rpc } +10s
Error: Error: P1002

The database server at `db.xxxxxxxx.supabase.co`:`5432` was reached but timed out.

Please try again.

Please make sure your database server is running at `db.xxxxxxxx.supabase.co`:`5432`.

    at Object.<anonymous> (/Users/ceb/projects/kana-api/node_modules/prisma/build/index.js:55736:26)
    at MigrateEngine.handleResponse (/Users/ceb/projects/kana-api/node_modules/prisma/build/index.js:55611:38)
    at LineStream.<anonymous> (/Users/ceb/projects/kana-api/node_modules/prisma/build/index.js:55696:18)
    at LineStream.emit (node:events:369:20)
    at LineStream.emit (node:domain:470:12)
    at addChunk (node:internal/streams/readable:313:12)
    at readableAddChunk (node:internal/streams/readable:288:9)
    at LineStream.Readable.push (node:internal/streams/readable:227:10)
    at LineStream._pushBuffer (/Users/ceb/projects/kana-api/node_modules/prisma/build/index.js:55451:19)
    at LineStream._transform (/Users/ceb/projects/kana-api/node_modules/prisma/build/index.js:55445:10)

Environment & setup
OS: Mac OS 10.15.4
Database: Postgres
Node.js version: v15.12.0
Prisma version:
prisma               : 2.21.2
@prisma/client       : 2.20.1
Current platform     : darwin
Query Engine         : query-engine e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/@prisma/engines/query-engine-darwin)
Migration Engine     : migration-engine-cli e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/@prisma/engines/migration-engine-darwin)
Introspection Engine : introspection-core e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/@prisma/engines/introspection-engine-darwin)
Format Binary        : prisma-fmt e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/@prisma/engines/prisma-fmt-darwin)
Default Engines Hash : e421996c87d5f3c8f7eeadd502d4ad402c89464d
Studio               : 0.371.0

What command exactly are you running @ChukaEbi? What kind of migration (and how much) would you expect that command to create and execute?

@janpio I am running npx prisma migrate dev and I expected it to just add a uniqueness constraint to an existing field in a table.

@ChukaEbi Are you able to reproduce this in the latest version?

I am still geting this bug with 2.26.

@meotimdihia Can you tell us where you are hosting your database?

@pantharshit00 I don't get this error anymore -  Prisma 2.27.

Closing due to inactivity

Issue persists... Running Windows 10 and connected Prisma to PostgreSQL running on Ubuntu 20.
Error: P1002

The database server at `<IP_ADDRESS>`:`5432` was reached but timed out.

Please try again.

Please make sure your database server is running at `<IP_ADDRESS>`:`5432`.

Context: Timed out trying to acquire a postgres advisory lock (SELECT pg_advisory_lock(72707369)). Elapsed: 10000ms. See https://pris.ly/d/migrate-advisory-locking for details.

Tried reinstalling node modules and undoing commits, nothing seems to work.

@s-kris Did you resolve this, i'm still consistently getting this, database is supabase as well

Same, this is happening at random intervals and just annoying. I am also using Supabase...

@gregorcee123 @mathiasklenk
I'm on version 2.27. This error doesn't seem to happen. Atleast not in the last six months.
My databases are on AWS RDS and Supabase(recent project).
I should also mention that my prisma + nextjs setup is a bit different.
https://superblog.ai/blog/using-prisma-with-nextjs-at-scale-ckpld2icd05251yo0mjow4fmi

@gregorcee123 @matt-development-work Are you also using migrate dev when you get this?
If not, please open a new issue and provide all the information you have so we can understand better. Thanks.

I'm getting this with npx prisma migrate dev. This just started happening to me after stepping away from the project I was using Prisma for.
$ docker exec -it nomads-api yarn migrate
yarn run v1.22.19
$ DEBUG=* npx prisma migrate dev
  prisma:engines  binaries to download libquery-engine, migration-engine, introspection-engine, prisma-fmt +0ms
  prisma:cache-dir  The following error is being caught and just there for debugging: +0ms
  prisma:cache-dir  [Error: EACCES: permission denied, mkdir '/root/.cache/prisma/master/d6e67a83f971b175a593ccc12e15c4a757f93ffe/linux-musl'] {
  errno: -13,
  code: 'EACCES',
  syscall: 'mkdir',
  path: '/root/.cache/prisma/master/d6e67a83f971b175a593ccc12e15c4a757f93ffe/linux-musl'
} +0ms
  prisma:cache-dir  The following error is being caught and just there for debugging: +3ms
  prisma:cache-dir  [Error: EACCES: permission denied, mkdir '/root/.cache/prisma/master/d6e67a83f971b175a593ccc12e15c4a757f93ffe/linux-musl'] {
  errno: -13,
  code: 'EACCES',
  syscall: 'mkdir',
  path: '/root/.cache/prisma/master/d6e67a83f971b175a593ccc12e15c4a757f93ffe/linux-musl'
} +0ms
  prisma:cache-dir  The following error is being caught and just there for debugging: +11ms
  prisma:cache-dir  [Error: EACCES: permission denied, mkdir '/root/.cache/prisma/master/d6e67a83f971b175a593ccc12e15c4a757f93ffe/linux-musl'] {
  errno: -13,
  code: 'EACCES',
  syscall: 'mkdir',
  path: '/root/.cache/prisma/master/d6e67a83f971b175a593ccc12e15c4a757f93ffe/linux-musl'
} +0ms
  prisma:cache-dir  The following error is being caught and just there for debugging: +5ms
  prisma:cache-dir  [Error: EACCES: permission denied, mkdir '/root/.cache/prisma/master/d6e67a83f971b175a593ccc12e15c4a757f93ffe/linux-musl'] {
  errno: -13,
  code: 'EACCES',
  syscall: 'mkdir',
  path: '/root/.cache/prisma/master/d6e67a83f971b175a593ccc12e15c4a757f93ffe/linux-musl'
} +0ms
  prisma:loadEnv  project root found at /var/www/api/package.json +0ms
  prisma:tryLoadEnv  Environment variables loaded from /var/www/api/.env +0ms
  prisma:getConfig  Using getConfig Wasm +0ms
  prisma:getConfig  config data retrieved without errors in getConfig Wasm +5ms
  prisma:loadEnv  project root found at /var/www/api/package.json +13ms
  prisma:tryLoadEnv  Environment variables loaded from /var/www/api/.env +12ms
Environment variables loaded from .env
Prisma schema loaded from prisma/schema.prisma
  prisma:getConfig  Using getConfig Wasm +4ms
  prisma:getConfig  config data retrieved without errors in getConfig Wasm +3ms
Datasource "db": PostgreSQL database "nomads", schema "public" at "pgsql:5432"

  prisma:getDMMF  Using CLI Query Engine (Node-API Library) at<EMAIL_ADDRESS>+0ms
  prisma:getDMMF  Loaded Node-API Library +0ms
  prisma:getDMMF  unserialized dmmf result ready +18ms
  prisma:getDMMF  dmmf retrieved without errors in getDmmfNodeAPI +5ms
  prisma:getConfig  Using getConfig Wasm +50ms
  prisma:getConfig  config data retrieved without errors in getConfig Wasm +2ms
  prisma:getConfig  Using getConfig Wasm +1ms
  prisma:getConfig  config data retrieved without errors in getConfig Wasm +1ms
  prisma:migrateEngine:rpc  starting migration engine with binary: /var/www/api/node_modules/@prisma/engines/migration-engine-linux-musl +0ms
  prisma:migrateEngine:rpc  SENDING RPC CALL {"id":1,"jsonrpc":"2.0","method":"devDiagnostic","params":{"migrationsDirectoryPath":"/var/www/api/prisma/migrations"}} +5ms
  prisma:migrateEngine:stderr  {"timestamp":"2023-03-02T21:12:13.248862382Z","level":"INFO","fields":{"message":"Starting migration engine RPC server","git_hash":"d6e67a83f971b175a593ccc12e15c4a757f93ffe"},"target":"migration_engine"} +0ms
  prisma:migrate:dev  {
  devDiagnostic: '{\n  "action": {\n    "tag": "createMigration"\n  }\n}'
} +0ms
  prisma:migrateEngine:rpc  SENDING RPC CALL {"id":2,"jsonrpc":"2.0","method":"applyMigrations","params":{"migrationsDirectoryPath":"/var/www/api/prisma/migrations"}} +764ms
  prisma:migrateEngine:rpc  {
  jsonrpc: '2.0',
  error: {
    code: 4466,
    message: 'An error happened. Check the data field for details.',
    data: {
      is_panic: false,
      message: 'The database server at `pgsql`:`5432` was reached but timed out.\n' +
        '\n' +
        'Please try again.\n' +
        '\n' +
        'Please make sure your database server is running at `pgsql`:`5432`.\n' +
        '\n' +
        'Context: Timed out trying to acquire a postgres advisory lock (SELECT pg_advisory_lock(72707369)). Elapsed: 10000ms. See https://pris.ly/d/migrate-advisory-locking for details.\n',
      meta: [Object],
      error_code: 'P1002'
    }
  },
  id: 2
} +10s
Error: Error: P1002

The database server at `pgsql`:`5432` was reached but timed out.

Please try again.

Please make sure your database server is running at `pgsql`:`5432`.

Context: Timed out trying to acquire a postgres advisory lock (SELECT pg_advisory_lock(72707369)). Elapsed: 10000ms. See https://pris.ly/d/migrate-advisory-locking for details.

    at Object.<anonymous> (/var/www/api/node_modules/prisma/build/index.js:93743:25)
    at MigrateEngine.handleResponse (/var/www/api/node_modules/prisma/build/index.js:93596:36)
    at LineStream3.<anonymous> (/var/www/api/node_modules/prisma/build/index.js:93695:16)
    at LineStream3.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Readable.push (node:internal/streams/readable:234:10)
    at LineStream3._pushBuffer (/var/www/api/node_modules/prisma/build/index.js:93473:17)
    at LineStream3._transform (/var/www/api/node_modules/prisma/build/index.js:93467:8)
    at Transform._write (node:internal/streams/transform:175:8)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I've got<EMAIL_ADDRESS>This is all running inside a docker setup which you can see here.

@simonv3 Same problem here, exactly like you described. :hmmm:

same

Resolved by enable PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK. Guess it is a network issue

Resolved by enable PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK. Guess it is a network issue

can you explain more ?

Resolved by enable PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK. Guess it is a network issue

This worked for me, don't know why though

Resolved by enable PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK. Guess it is a network issue

Thank you , this worked.
For local, you can add the value in your .env file
PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK=true
For gitlab pipeline, you can update .gitlab-ci.yml with
    variables:
        PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK: true

For local, you can add the value in your .env file
PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK=true

Confirming that this is still working in 2024. Helped unblock my dockerized local development issues, running:
docker exec container-name npx prisma migrate dev --name latest
The weird thing is that it ran totally fine a few times before without needing this .env value, but now won't work without it.


Release Checklist v25.02

Release Checklist for v25.02:

Status
Task
Start
End
Owner

Pre-release

- [ ]
Notification to team of last feature sprint in milestone
2025-01-20
2024-01-31
@mattahrens

- [ ]
Burndown (no new features for current release)
2025-02-02
2025-02-14
Team

- [ ]
Update docs for current release in spark-rapids without updating release links
2025-02-10
2025-02-14
Team

- [ ]
Create a PR to update release links
2025-02-14
2025-02-14
@nvliyuan

- [ ]
Get RAPIDS 25.04 cudf java build working.
2025-01-27
2025-01-31
@NvTimLiu

- [ ]
Create 25.04 branch including spark-rapids-{jni,private} and spark-rapids
2025-01-20
2025-01-24
@NvTimLiu

- [ ]
Automerge from 25.02 to 25.04
2025-01-20
2025-01-24
@NvTimLiu

- [ ]
Enable Jenkins builds (premerge, nightly, integration, performance) for 25.04
2025-01-20
2025-01-24
@NvTimLiu

- [ ]
[spark-rapids-private 25.02] Java release to sonatype
2025-02-14
2025-02-14
@NvTimLiu

- [ ]
[spark-rapids-jni 25.02] Java release to sonatype
2025-02-14
2025-02-14
@NvTimLiu

- [ ]
[cudf 25.02] Java release to sonatype
2025-02-14
2025-02-14
@NvTimLiu

- [ ]
Notify RAPIDS team of javadocs for 25.02
2025-02-17
2025-02-17
@GaryShen2008

Staging release

- [ ]
Make 25.04 the default branch
2025-02-10
2025-02-10
@jlowe

- [ ]
Enable static scanning on the new 25.04 branches
2025-02-20
2025-02-20
@GaryShen2008

- [ ]
Merge all remaining branch-25.02 PRs
2025-02-14
2025-02-14
Team

- [ ]
Update spark-rapids-jni dependency version
2025-02-14
2025-02-14
@NvTimLiu

- [ ]
Create CHANGELOG.md
2025-02-14
2025-02-14
@NvTimLiu

- [ ]
Deliver Snapshot packages to QA
2025-02-10
2025-02-10
@NvTimLiu

- [ ]
Create a PR (not squashed) to merge from branch-25.02 to main. Needs review from repository owner
2025-02-14
2025-02-14
@NvTimLiu

- [ ]
Trigger the release for spark-rapids Jenkins job on the main branch and deploy the jar to sonatype staging (including spark-rapids-tools/core)
2025-02-18
2025-02-18
@NvTimLiu

- [ ]
Deliver final packages to QA
2025-02-18
2025-02-18
@NvTimLiu

- [ ]
Run integration tests on staging jars with AQE on, UCX on, Databricks Azure, Databricks AWS, Dataproc
2025-02-18
2025-02-21
@NvTimLiu

- [ ]
Run LHA tests on staging jars
2025-02-18
2025-02-21
@NvTimLiu

- [ ]
CDH testing(skip)
2025-02-18
2025-02-20
@NVTimLiu

- [ ]
QA testing with Snapshot on YARN, NGC with Volta and Ampere
2025-02-10
2025-02-18
@johnzhong

- [ ]
~~Run pre-processing workflow for DLRM sample use case
2025-02-14
2025-02-18
@johnzhong

- [ ]
QA testing with final packages on YARN, NGC with Volta and Ampere
2025-02-18
2025-02-21
@johnzhong

Release

- [ ]
Tag the release and deploy the staging artifacts to maven central repository
2025-02-26
2025-02-26
@NvTimLiu

- [ ]
Update gh-pages
2025-02-17
2025-02-26
@nvliyuan

- [ ]
Test spark-rapids-examples repo with Spark-Rapids 25.02
2025-02-17
2025-02-26
@nvliyuan

- [ ]
Update docs for current release in spark-rapids-examples
2025-02-17
2025-02-26
@nvliyuan

- [ ]
Update tools notebooks on Databricks and EMR for current release in spark-rapids-examples
2025-02-17
2025-02-26
@nvliyuan

- [ ]
Update spark-rapids.sh in https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/spark-rapids for the plugin
2025-02-17
2025-02-26
@nvliyuan

To avoid the overlap of Chinese New Year Holiday, update the Start/End Dates to (2025-01-20 to 2025-01-24) for the tasks related to creating branch-25.02 and setting up CI/CD for the rapids-JNI, rapids-private, and spark-rapids repos.


eval-less implementation, preliminary to browser support

On some browser-python runtimes, eval() is either catastrophically slow (brython, skulpt) or completely unavailable (pyj(ama)s, batavia).  Since attrs is The Library You Should Absolutely Use No Matter What™, it would be nice if it had an affordance to support those runtimes.

We don’t use eval! :sparkle: :D
I guess exec is evil too?

Oh. It probably should use eval :).

But yes, exec has exactly the same limitations.

See #87.

This might be an interesting technical challenge. But how do we go about it operationally?

how to separate the eval code from the fallbacks?
how to test the fallbacks?

@Tinche First - we don't need to do all of those in the same PR :).

how to separate the eval code from the fallbacks?

They should just be in totally separate functions.  Not supporting eval efficiently is a reasonable thing for a Python runtime to do given certain constraints; but everybody can support returning a higher-order function, so we just have two ways to construct the __init__.

how to test the fallbacks?

This doesn't seem hard to me; just set a flag and run all the same unit tests to ensure the behavior is the same.

This doesn't seem hard to me; just set a flag and run all the same unit tests to ensure the behavior is the same.

Ideally we could test using one of the runtimes you enumerated. Or monkeypatch eval to blow up when called, but that might break other libs like pytest?

Ideally we could test using one of the runtimes you enumerated. Or monkeypatch eval to blow up when called, but that might break other libs like pytest?

For testing purposes, we can have a private API that explicitly selects the init-creation mode; @attr._s_but_with_no_eval, as a strawman example.

What's the alternative to using eval/exec?  For CPython and pypy, we could emit bytecode directly, but that also doesn't support every python implementation out there (pyston).

What's the alternative to using eval/exec?

Calling setattr.

Here's another use case for eval-less class creation: building classes from configuration such as swagger/openapi. Because the spec might come from an untrusted source, eval is ill-advised.

Because the spec might come from an untrusted source

If this is the case, then the swagger loader ought to be properly parsing and sanitizing everything (transforming everything into valid identifiers, for example).  If the inputs are in the appropriate format, then eval should be just as safe as any other construction mechanism.

Just bumped into this issue - our environment doesn't allow either eval or compile. Is there any way we can use this package without either of those functions being required?

Not at this time. That should be possible once we have pluggable method builders but there’s sadly no timeline.

@tyteen4a03 I am super curious—what environment is this?

Not at liberty to say unfortunately, but we do have good reasons for this.

@tyteen4a03 Understandable — independent of any reason, any interest in submitting a PR? :)


vdk-core: add config option for logging execution result

Why?
User reasearch indicates that the execution result is too verbose for local runs. Users don't expect a lot of output for successful jobs and expect error output for failing jobs.
What?
Introduce the LOG_EXECUTION_RESULT config option that enables/disables displaying the end result.
Config Error Output
https://pastebin.com/raw/MvJX0u9m
User Error Output
https://pastebin.com/raw/0MxZp6Zg
Success Output
https://pastebin.com/raw/GTNBiEFG
How was this tested?
Ran locally with successful and failing jobs
What kind of change is this?
Feature/non-breaking

Please add tests?
I think the easiest way to test them would be with functional test:
    runner = CliEntryBasedTestRunner()

    result: Result = runner.invoke(["run", util.job_path("fail-job-indirect-library")])

the result object contain stderr and std out which you can inspect for verification.


Google code shutting down is causing 502 errors during Travis builds

Several recent builds have failed as a result of Travis being unable to resolve artifacts delivered via Google Code repositories - in particular the version of ZXing for QR code creation.
As a related issue our GitHub based staging repo is unreliable in Intellij resulting in the need for manual builds from source to take place. Changing the URL resolver to https://rawgithub.com should fix these issues.

Ready for review and close. To verify delete or rename the following in your local repo:

org/bitcoinj/bitcoinj-core/0.13-alice-develop-SNAPSHOT - (rawgithub.com)
org/multibit/hd/hardware/ - (rawgithub.com)
com/google/zxing - (Maven Central)

Attempt "Rebuild All" within Intellij (it might fail). A Maven build should restore them correctly.
Ready for review and close.

Delete the artifacts mentioned and rebuilt usng 'mvn clean install'. I could see it pulling in the artifacts and it built fine.
Closing.


to silence all the output from webdriver(chromedriver) for log level INFO

All output will show after specifying WECHATY_LOG=verbose or silly.
This will make more users happy because wechaty should not output so many debug info unless user want to see them.

@lijiarui Could you please help me to confirm whether this commit could disable all the webdriver log message when using the default setting of `WECHATY_LOG=info'?
Thanks.


Why does this project exist?

Shouldn't it be better to use https://github.com/OpenZeppelin/openzeppelin-solidity/tree/master/contracts/token/ERC721 ?

Thanks for the question @eordano. This implementation is 100% aligned with the ERC-721 standard. Other implementations (currently) are broken in some places and do not fully respect the standard.


Caching latents with multiple GPUs

awesome. this will work automatically when multi gpu used?

@kohya-ss those --highvram and --lowvram made 0 impact on my testings previously what they do actually? I tested both for FLUX Fine tuning and FLUX LoRA training
I can test FLUX LoRA multi gpu caching - fine tuning still requires 80 gb GPUs, fused backward pass not working

@kohya-ss you are awesome thank you so much

I hope the full version of GPT 01 will help find the right solution very soon.

I hope it is possible to push the cached data directly to HF, so that pulling latents directly in the cloud platform will not take up cache time, and more hard disk space can be saved on large datasets.


makes the various timeouts configurable on the connection -- connect,…

… query, transaction
We have a need to configure these settings. Also, we have a need to disable the timeouts in some cases -- such as database migrations which may take a long time. Therefore, if the configured timeout is set to 0 it is ignored createContext

I didn't see any relevant tests for these and the changes are pretty straightforward, so did not create any

Thanks for thinking about this.
AFAIK,
Clients can any control queryTimeout.
Transaction timeout  is configured on VtTablets.
Connection timeout is not properly.
Only change we can do in clients is about queryTimeout.

Reviewed 4 of 7 files at r1.
Review status: 4 of 7 files reviewed at latest revision, 1 unresolved discussion.

java/jdbc/src/main/java/io/vitess/jdbc/ConnectionProperties.java, line 210 at r1 (raw file):

        true);

    private LongConnectionProperty queryTimeoutMillis = new LongConnectionProperty(

I think we can remove millis from the property and just mention in the property description.

Comments from Reviewable

I'm away for the next week if any further comments come up. But hopefully this should be good to go.

I meant, we can just have queryTimeout property for now and use it everywhere, till we do not support other property end to end.

Reviewed 3 of 7 files at r1, 5 of 5 files at r2.
Review status: all files reviewed at latest revision, 2 unresolved discussions.

java/jdbc/src/main/java/io/vitess/util/CommonUtils.java, line 36 at r1 (raw file):

     * @return
     */
    public static Context createContext(String username, long connectionTimeout) {

we can rename connectionTimeout to timeout. As it is only query execution timeout.

Comments from Reviewable

@harshit-gangal ok sounds good. I'll unify everything on a single queryTimeout config.  I'll try to do that tomorrow morning so we can hopefully merge before I leave for the week.

@harshit-gangal ok I renamed to timeout and unified the properties on just the 1

Reviewed 8 of 8 files at r3.
Review status: all files reviewed at latest revision, 2 unresolved discussions.

Comments from Reviewable

LGTM


Fixed several Python 3 compatibility problems

using sys.version_info to be backwards compatible
map is now lazy
range does what xrange did before
tested changes with 2.7, 3.4 and 3.5 (shouldn't matter)

Python3 compatibility has now been added, although the performance optimizations checking python/python3 versions have been eschewed in favor of simple code.


Start to document the details of vararg handling

See also issue #29. I expect the description in this PR is not yet clear or complete enough - feedback very welcome.

Thanks, it's a good start


1.2.2 "ionic serve" just hangs and not responding

Ionic 2
app-scripts 1.2.2
After I updated ionic-app-scripts from 1.1.4 to the latest 1.2.2, ionic serve just hangs without any error message.
I tried setting up a new project with ionic start testIonic --v2 and experienced the same behavior.

@anton-white,
That seems strange to me. How can I recreate it?
Thanks,
Dan

@danbucholtz
$ npm list -g --depth=0
/usr/lib
├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>└──<EMAIL_ADDRESS>

ionic start testIonic --v2
cd testIonic
ionic serve

I can confirm the behavior, just hangs after ionic serve. (Ubuntu)

Me too on Ubuntu 16.04 desktop

I have downgraded the version of app-scripts to 1.1.4 with npm install<EMAIL_ADDRESS>--save-dev. Now it works

Man, that's cool :)
but if you look closer at my initial message:

After I updated ionic-app-scripts from 1.1.4 to the latest 1.2.2

This is the issue right now, there are few issues open at ionic-cli repo, the fix so far is to downgrade it back to 1.1.4 as suggested by @pawelkat. However, I think we are looking for the long term fix, which is fix in the latest script rather than downgrade.

i have the same problem ... fix it with downgrading
npm install<EMAIL_ADDRESS>--save-dev

i have the same problem ... fix it with downgrading
npm install<EMAIL_ADDRESS>--save-dev

Are you all on linux? There is a known issue fixed in nightly for linux.
Thanks,
Dan

@danbucholtz  no, windows 7
<EMAIL_ADDRESS>up all versions not working,
1.1.4  works

This should be working now with 1.2.3. Can you please let me know if it's not and I'll re-open.
Thanks,
Dan

@danbucholtz,
it's working now. Thanks a lot!


WIP: Enhance Case Management - implement batch fetches for multiple entities

https://github.com/Gizra/ihangane/issues/3066

Before: 1352 requests:

After: 12 requests:


Which auras require crafted item

Hello, appreciate the work!
I was wondering if you planned to add something that indicated if an aura needed a crafted item to be enabled.
I was thinking maybe an additional field underneath aura

It sounds like a good idea

@mlg404
Do you have any plans to add a Craft item list?

@mlg404
Do you have any plans to add a Craft item list?

Yes! I just need some time to do that!

Hello, appreciate the work! I was wondering if you planned to add something that indicated if an aura needed a crafted item to be enabled. I was thinking maybe an additional field underneath aura

I just added support for this, but my info source has only the FoxParks tech (Tech Needed)
I get info (automatic) from this wiki https://palworld.fandom.com/wiki/Foxparks
When all Pal's get theirs respectives techs, i update this issue branch with fresh data

nevermind
i just added all info to the wiki by myself and scraped it all too

PR #12
BRANCH #16
Just merged it to main.
i closed the wrong issue so i'm commenting this


Add ENOATTR for Linux

It's defined differently for OSX and *BSD, so having a Linux definition is helpful for cross-platform code.

The #include <attr/xattr.h> seems more complicated than expected. It's defined by libattr, which we don't have on all platforms. That's not necessary for the definition itself though, just the tests. How should I handle this? Just disable the tests for the constant?

Also, this change is failing the linter as the constant definition comes at the end, but I think it needs to be there to use the platform specific ENODATA definition. Once again, how should I handle this?

Oh right now we don't actually test uclibc on CI yet, so it's ok to back out those changes if you'd like. Also no need to worry about definition order, it doesn't have to literally come after ENODATA, the definitions can be in either order.

How should I deal with needing the attr headers for tests? They aren't present on some test systems.

Whatever gets past CI is fine by me, I don't particularly have opinions how it gets past CI

Do you mind if I just ignore the constant in the tests? The setxattr and getxattr man pages on linux specifically say EATTR == ENODATA.

Hm man pages typically tend to be right for one architecture and I've seen they don't always tend to be right across all of them. Do you know why this header can't be found?

It comes from libattr, which we don't have for a lot of Linux platforms.

Because of that, maybe it shouldn't be in this Rust crate, but I think it's very important as other platforms define it and it's explicitly mentioned in the man page.

From the libattr source: http://download.savannah.gnu.org/releases/attr/attr-2.4.47.src.tar.gz
include/xattr.h:
#include <errno.h>
#ifndef ENOATTR
# define ENOATTR ENODATA        /* No such attribute */
#endif

If defined, ENOATTR will always be an alias for ENODATA on Linux.

If this constant is only available on architecture, perhaps it could just be included there? The package should be installable via the various docker images we've got.

Okay, I'll see if I can install them in docker.
ENOATTR is available for all architectures. It's just that libattr isn't installed by default on all platforms.

Hm if this isn't actually available in libc itself but requires an extra package to be installed, should it be included in the libc crate itself?

I'm not sure. It is in libc for other platforms, so it's useful for cross-platform code. It's also mentioned in the setxattr and getxattr man pages (which don't come from libattr but Linux).

Ok, well it sounds like the story here is fishy enough that it seems safe to just skip the constant, mind adding these comments to a comment as to why we're skipping it?

Done. I'm assuming we don't want rustdoc comments for the constant definition? We don't have them for any other constants.

:pushpin: Commit 98889cf has been approved by alexcrichton

Nah that's ok, we typically don't have lots of rustdoc docs in libc (we rely on libc itself documentation for that)
@bors: r+

:hourglass: Testing commit 98889cf with merge a3e79eb...

:sunny: Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing a3e79eb110465eaedc55a7df508ca1e4604d1b5d to master...


Projection with LokiJS to show specified fields only

Can projection be used with LokiJS to show specified fields only?
Simular to https://docs.mongodb.com/manual/reference/method/db.collection.find/#find-projection
Thanks!

I think you'd have to do a javascript array map() on the results...
in theory we could add our own map or project function which accept an object to control fields / mapping... if we did that you could persist that (along with the database) if using in collection transforms


Recent Posts section on index deployment does not match local

Environment

Minimal Mistakes version: minimal-mistakes-jekyll 4.16.0
Ruby gem or remote theme version: 2.7.6
Jekyll version: jekyll 3.8.5
Git repository URL: https://github.com/kicksent/kicksent.github.io
GitHub Pages hosted (if yes provide URL to site): https://kicksent.github.io
Operating system: Windows 10 local env running through Ubuntu App

Expected behavior
Expected LOCAL: 1. Recent Posts on Index 2. Recent Posts at bottom of post
Actual LIVE: 1. Recent Posts on Index 2. Recent Posts at bottom of post

Steps to reproduce the behavior

fork my repo and serve with bundle exec jekyll serve
(I was able to reproduce the issue like this using an alternate github account and hosting it on github pages on the other github account)
compare to site at kicksent.github.io

Other

Closing and rebuilding site using minimal as remote theme.


Upgrade goclient library and fix linting

Description
Upgrade goclient library and fix linting.
GitHub Issues
List the GitHub issues impacted by this PR:

GitHub Issue #

#https://github.com/dell/csm/issues/1098

Checklist:

[x] I have performed a self-review of my own code to ensure there are no formatting, vetting, linting, or security issues
[x] I have verified that new and existing unit tests pass locally with my changes
[x] I have not allowed coverage numbers to degenerate
[x] I have maintained at least 90% code coverage
[ ] I have inspected the Grafana dashboards to verify the data is displayed properly
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[x] I have maintained backward compatibility

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Please also list any relevant details for your test configuration

[ ] Test A
[ ] Test B

Manual inspection of the GUI
I have verified that the dashboards show the data properly while generating I/O and storage resources

[ ] Yes
[ ] No

e2e pass

e2e pass

e2e pass


Separate the usage section

Thanks, @J2TeaM . If you want to add your name to the contributors array in package.json, go ahead with another commit before I merge ;)

@hiendv Done!! :+1:


App freezes for a while when there are a lot of annotations

I have deployed Beaverdam on a public cloud, and I am using it to annotate multiple objects.
When there are a lot of annotations on a video, the app freezes for a while (3-4 seconds) every time I try to add or resize an annotation.
I suspect it is because of the GUI elements (keyframebar, annotationbar, svg elements) being continuously reloaded after every action.
Any idea how to fix this?

Hmm, how long is your video? And how many annotations (visible and hidden) are we talking about?
@szhu

My videos are between 5 and 10 minutes long.
The interface is freezing for a video that has around 2000 annotations (individual keyframes).

Can you use Chrome's DevTools and capture a profile of the slow load? In DevTools, click the Performance tab then click record before the freeze happens, then stop recording after the page is responsive again. Then the Bottom-up and Call Tree tabs on the bottom can help us figure out what the slow functions are.
You can attach a screenshot of both profiler tabs and/or save the profile and attach a link to that. If you're looking for a free way to share the profile link, you can try using CloudApp -- I think it's free.

It's probably because the js does a for loop search through all the keyframe comparing the time of each keyframe to the current video time +/- some delta to find the current key frame as well as reinserting the keyframe when updating it. You can probably verify this if the problem only occurs when you edit annotations that are later in the video compared to editing annotations in the beginning of the video.

I've always annotated videos in <1min chunks. I can see how having that many annotations could cause performance issues. Did you try running it through the sampling profiler?

@szhu Here is the profile for when I move an existing annotation.
profile.txt
@clayshieh I notice the same lag whether it is at the beginning or at the end of the video. I believe it's  because the following handlers are triggered for every action, therefore the json data containing the annotations is getting parsed and processed over and over again:
$(this).triggerHandler('change-onscreen-annotations');
$(this).triggerHandler('change-keyframes');

Thank you both for your replies.

Sorry, I can't load your profile for some reason. I get a blank "Performance" tab with no error message, even when running the server from localhost:9229 like you did. You can try one of the following:

Attaching screenshots of the relevant tabs in the Performance tool
Try to figure out Performance tool works to figure out which function(s) are taking a long time
Attaching a set of annotations (I forget how to export them) and a video that can be used to reproduce this issue

@szhu Thank you once again for your reply. I had not deployed Beaverdam on localhost:9229 but on a private IP address with port 5000.
Here are some screenshots of the Performance tool, and I have also attached another profile corresponding to these screenshots.

profile-20180316.txt
Much appreciated.


Update Brunch instructions (postcss-brunch)

Since v0.3.0 postcss-brunch has the new syntax.

Thanks @dmitrykiselyov


Typo in graphic 1-2-a

A text was copied with comment:

Replace with "No prominent sharing of the story URL on public Facebook pages and groups from late May to early November according to CrowdTangle data

done!


"Received 'killed' signal" during the circleci python3 build after submitting PR

I submitted a PR after modifying convert_gpt2_checkpoint_to_pytorch.py, and my build_py2 test passed, but I received a very vague error from build_py3 (as written in the title of this issue) that caused my build to fail. Does anyone have any ideas as to where the issue could be?
Edit: Attached image below

Yeah I've removed the memory-heavy tests

Thanks!


John's updated files

You should be able to download 60 FCS files with “Q2LabSol_Aurora-2” in the name.  Please replace the existing “Q2LabSol_Aurora-2” FCS files in LabCAS with these files (they have the same names).  The metadata tables do not change.
This is left over from the problem that a company had two flow cytometers of the same name and it messed up our conversions.  Ideally this should fix it.

@yuliujpl Looks great!


Aux window: drag/drop tab onto desktop works but shows a "no drop" pointer

Type: Bug
Testing #8171 on Windows 11, I was disconcerted to see the "no drop" pointer (red circle, red diagonal slash through it). But when I dropped, the aux window opened correctly.
VS Code version: Code - Insiders 1.85.0-insider (345c8d0927402d41f42402411d856eeec4013e5e, 2023-11-13T05:39:23.476Z)
OS version: Windows_NT x64 10.0.22621
Modes:

/assign @bpasero

Unfortunately I do not see a way to address this without causing unwanted side effects on the target application that you are dropping, but if someone has suggestions, please speak up.
Essentially I want to indicate that dropping is possible but without the target application to accept the data (e.g. a terminal should not change the prompt with the data of the dragged tab).

Btw I cannot reproduce the "no drop" pointer when dropping over desktop:

Desktop still gives me the no-drop indicator using this morning's Insiders:

It was Insiders.

On macOS this is even worse because there is an ugly 500-1000ms animation for invalid drops that causes all sorts of issues:

the entire operation feels sluggy and slow
if you manage to move the mouse back over the main window shortly after releasing, we cancel the operation assuming you wanted to drop over the main window


Do not --prefer-source for composer update

Like https://github.com/sabre-io/skel/pull/13

There are actually no unit tests in this repo, so this PR does not make a difference - no tests are run!

Love the pull request on a web finger


Allow to create group names with spaces

Description:
As a Happa user, I would like to create group names with spaces.
Currently, space is the delimiter on group names. Therefore the group name with space is split into different groups.
Actions:

To remove the space delimitation and chose another delimiter (e.g , or ; )
Update the message to indicate multiple groups can be created with the selected delimiter

Just ran into this limitation while demoing role binding creation on icecream, which has group names with spaces in them. I think with the changes to support making these bindings easily from happa already released with rbac-operator, this should be worked on next cc @PrashantR31 for next week's planning


Approximation or search in a table with parabolic interpolation?

@faf0, in version 1.4, an approximation of the form is used to determine the current temperature:
t = a0 + a1r Gr + a2r Gr^2 + a1g Gg + a2g Gg^2 + a1b Gb + a2b Gb^2

https://github.com/faf0/sct/blob/a38a0df3f85e0c437de3a28cf1627bcae68ef4ff/sct.c#L92-L94
Its results are as follows:
$ for i in $(seq 10); do echo "T = $(expr $i \* 1000)"; ./xsct $(expr $i \* 1000) && ./xsct; doneT = 1000
Screen 0: temperature ~ 1064
T = 2000
Screen 0: temperature ~ 2075
T = 3000
Screen 0: temperature ~ 3012
T = 4000
Screen 0: temperature ~ 4068
T = 5000
Screen 0: temperature ~ 4988
T = 6000
Screen 0: temperature ~ 6029
T = 7000
Screen 0: temperature ~ 6951
T = 8000
Screen 0: temperature ~ 8037
T = 9000
Screen 0: temperature ~ 9109
T = 10000
Screen 0: temperature ~ 10066

Alternative - search in a table with parabolic interpolation:
#define TEMPERATURE_COUNT   20
#define DIST(gr,gg,gb,i) ((gr) - whitepoints[i].r) * ((gr) - whitepoints[i].r) + ((gg) - whitepoints[i].g) * ((gg) - whitepoints[i].g) + ((gb) - whitepoints[i].b) * ((gb) - whitepoints[i].b)

static int get_sct_for_screen(Display *dpy, int screen)
{
    Window root = RootWindow(dpy, screen);
    XRRScreenResources *res = XRRGetScreenResourcesCurrent(dpy, root);

    int temp = 0, n, c, m, offset = 1;
    double t = 0.0;
    double gammar = 0.0, gammag = 0.0, gammab = 0.0, gammam = 1.0;
    double mingd, tgd, gdA, gdB, a1, a2, dm;

    n = res->ncrtc;
    for (c = 0; c < n; c++)
    {
        int crtcxid = res->crtcs[c];
        XRRCrtcGamma *crtc_gamma = XRRGetCrtcGamma(dpy, crtcxid);
        int size = crtc_gamma->size;
        gammar += (crtc_gamma->red[size - 1]);
        gammag += (crtc_gamma->green[size - 1]);
        gammab += (crtc_gamma->blue[size - 1]);

        XFree(crtc_gamma);
    }
    gammam = (gammar > gammag) ? gammar : gammag;
    gammam = (gammab > gammam) ? gammab : gammam;
    if (gammam > 0)
    {
        gammar /= gammam;
        gammag /= gammam;
        gammab /= gammam;
        m = offset;
        mingd = DIST(gammar, gammag, gammab, m);
        for (c = offset; c < TEMPERATURE_COUNT - offset; c++)
        {
            tgd = DIST(gammar, gammag, gammab, c);
            if (tgd < mingd)
            {
                m = c;
                mingd = tgd;
            }
        }
        gdA = DIST(gammar, gammag, gammab, m - offset);
        gdB = DIST(gammar, gammag, gammab, m + offset);
        a1 = (gdB - gdA) * 0.5;
        a2 = (gdB + gdA) * 0.5 - mingd;
        dm = (a2 == 0.0) ? 0.0 : (-0.5 * a1 / a2);
        dm += (double)m;
        t = dm * (double)TEMPERATURE_STEP;
        t += (double)TEMPERATURE_MIN;
    }

    temp = (int)(t + 0.5);

    XFree(res);
    return temp;
}

static void sct_for_screen(Display *dpy, int screen, int temp)
{
    Window root = RootWindow(dpy, screen);
    XRRScreenResources *res = XRRGetScreenResourcesCurrent(dpy, root);

    temp -= TEMPERATURE_MIN;
    double ratio = (double)(temp % TEMPERATURE_STEP) / TEMPERATURE_STEP;

Its results are as follows:
$ for i in $(seq 10); do echo "T = $(expr $i \* 1000)"; ./xsct $(expr $i \* 1000) && ./xsct; done
T = 1000
Screen 0: temperature ~ 1113
T = 2000
Screen 0: temperature ~ 2082
T = 3000
Screen 0: temperature ~ 3039
T = 4000
Screen 0: temperature ~ 4035
T = 5000
Screen 0: temperature ~ 5021
T = 6000
Screen 0: temperature ~ 6028
T = 7000
Screen 0: temperature ~ 7045
T = 8000
Screen 0: temperature ~ 8039
T = 9000
Screen 0: temperature ~ 9035
T = 10000
Screen 0: temperature ~ 9945

But at the same time, the function does not contain "incomprehensible" coefficients.
I tend to search the table.

Thanks for the suggestion. I don't have strong opinion here with a slight tendency to search the table.

PR #10


Added my Contribution Data.

Props for the maintainers!!!!

Please accept my contributions for Hacktoberfest22.

Good job @muhammad-mobeen
Congrats 🎉, your PR got merged. To be eligible for the swags from ClueLess. Follow this steps.
Note that this repo is excluded from Hacktoberfest but this PR is still eligible for ClueLess Swags.

Star this repository. ⭐
SignIn / SignUp in [ClueLess Website](https://www.clueless.tech/) .
Go to profile section from the top right hand corner.

Comment your profile URL under this comment tagging the person who merged your PR.

That’s it. Follow the leaderboard !
And don’t forget to share your achievement with your friends.

@Rajdip019
Profile URL: https://www.clueless.tech/profile/admin/117472736979105215173


Set Up a R/O Shared Set of Unmodified Class Materials

Could someone copy an unmodified set of Class Materials to a separate High Level Qualifier that all users on the system would have R/O access to, such as 'SYS2.Z80000.CBL', 'SYS2.Z80000.JCL', etc.?  This would serve serveral purposes:

Have a clean set of files (already loaded on the mainframe) to restore from, in case an individual user "destroys" their copy.

Have a place to put additional utility JCL jobs that are created by Admins in response to questions asked in the (Slack, etc.) forums, such as DEFVSAM, as discussed here: https://openmainframeproject.slack.com/archives/C011NE32Z1T/p1587999987108000?thread_ts=1587936917.072900&cid=C011NE32Z1T

Have access to a set of R/O PDS, SEQ and DATA files for the purpose of practicing copying them, using standard IBM utilities, like IEBGENER, SORTGENR, IEBCOPY, etc.

FYI, if granted sufficient RACF authority, I would be glad to take on this task.
Thank you,
Scott

Unmodified labs data sets are available to all in a read only state
ZOS.PUBLIC.CBL
ZOS.PUBLIC.DATA
ZOS.PUBLIC.JCL
ZOS.PUBLIC.LOAD
ZOS.PUBLIC.SEQ.DATA
ZOS.PUBLIC.JCL(DEFVSAM) defines VSAM KSDS and copies DATA file content in the VSAM.KSDS

I can create a single ZOS.PUBLIC.SHARED.JCL (public read/write) for any to add JCL members

@paulnewt Maybe you would want to keep the public files exactly as they were when originally created and create another PDS, say, ZOS.PUBLIC.ADHOC.JCL which contains added JCL beyond the course materials?  Just a thought!  Thanks!

Data set allocated with public update access ----
ZOS.PUBLIC.ADHOC.JCL($README)
Member names with brief description
DEFVSAM - DEFINE VSAM KSDS & REPRO DATA consisting of a unique key


fluent/lib/ruby/3.2.0/win32/registry.rb:910:in `encode': U+767E to ASCII-8BIT in conversion from UTF-16LE to UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError)

Describe the bug
Windows 11  run fluentd  error：
D:/opt/fluent/lib/ruby/3.2.0/rubygems.rb:1342:in rescue in <top (required)>': U+767E to ASCII-8BIT in conversion from UTF-16LE to UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError) Loading the D:/opt/fluent/lib/ruby/3.2.0/rubygems/defaults/operating_system.rb file caused an error. This file is owned by your OS, not by rubygems upstream. Please find out which OS package this file belongs to and follow the guidelines from your OS to report the problem and ask for help.         from D:/opt/fluent/lib/ruby/3.2.0/rubygems.rb:1328:in <top (required)>'
from internal:gem_prelude:2:in require'         from <internal:gem_prelude>:2:in internal:gem_prelude'
D:/opt/fluent/lib/ruby/3.2.0/win32/registry.rb:910:in encode': U+767E to ASCII-8BIT in conversion from UTF-16LE to UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError)         from D:/opt/fluent/lib/ruby/3.2.0/win32/registry.rb:910:in export_string'
from D:/opt/fluent/lib/ruby/3.2.0/win32/registry.rb:611:in each_key'         from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:71:in block (2 levels) in iterate_msys_paths'
from D:/opt/fluent/lib/ruby/3.2.0/win32/registry.rb:435:in open'         from D:/opt/fluent/lib/ruby/3.2.0/win32/registry.rb:542:in open'
from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:70:in block in iterate_msys_paths'         from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:68:in each'
from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:68:in iterate_msys_paths'         from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:102:in msys_path'
from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:115:in mingw_bin_path'         from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:125:in enable_dll_search_paths'
from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/singleton.rb:27:in enable_dll_search_paths'         from D:/opt/fluent/lib/ruby/3.2.0/rubygems/defaults/operating_system.rb:24:in <top (required)>'
from D:/opt/fluent/lib/ruby/3.2.0/rubygems.rb:1332:in require'         from D:/opt/fluent/lib/ruby/3.2.0/rubygems.rb:1332:in <top (required)>'
from internal:gem_prelude:2:in require'         from <internal:gem_prelude>:2:in internal:gem_prelude'
2024-02-28 14:24:25 +0800 [error]: Worker 0 exited unexpectedly with status 1
D:/opt/fluent/lib/ruby/3.2.0/rubygems/specification.rb:10:in require_relative': Interrupt         from D:/opt/fluent/lib/ruby/3.2.0/rubygems/specification.rb:10:in <top (required)>'
from D:/opt/fluent/lib/ruby/3.2.0/rubygems.rb:1325:in require_relative'         from D:/opt/fluent/lib/ruby/3.2.0/rubygems.rb:1325:in <top (required)>'
from internal:gem_prelude:2:in require'         from <internal:gem_prelude>:2:in internal:gem_prelude'
To Reproduce
Initial installation
Expected behavior
normal running load
Your Environment
2024-02-28 14:34:34 +0800 [info]: fluent/log.rb:362:info: init supervisor logger path=nil rotate_age=30 rotate_size=nil
2024-02-28 14:34:34 +0800 [info]: fluent/log.rb:362:info: parsing config file is succeeded path="D:/opt/fluent//etc/fluent/fluentd.conf"
2024-02-28 14:34:34 +0800 [info]: fluent/log.rb:362:info: gem 'fluentd' version '1.16.3'
2024-02-28 14:34:34 +0800 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-calyptia-monitoring' version '0.1.3'
2024-02-28 14:34:34 +0800 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-elasticsearch' version '5.4.0'

Your Configuration
<source>
  @type windows_eventlog2
  @id windows_eventlog2
  channels application
  read_existing_events false
  tag winevt.raw
  rate_limit 200
  <storage>
    @type local
    persistent true
    path D:\opt\fluent\winlog.json
  </storage>
</source>

<match winevt.raw>
  @type stdout
</match>

Your Error Log
D:/opt/fluent/lib/ruby/3.2.0/rubygems.rb:1342:in `rescue in <top (required)>': U+767E to ASCII-8BIT in conversion from UTF-16LE to UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError)
Loading the D:/opt/fluent/lib/ruby/3.2.0/rubygems/defaults/operating_system.rb file caused an error. This file is owned by your OS, not by rubygems upstream. Please find out which OS package this file belongs to and follow the guidelines from your OS to report the problem and ask for help.
        from D:/opt/fluent/lib/ruby/3.2.0/rubygems.rb:1328:in `<top (required)>'
        from <internal:gem_prelude>:2:in `require'
        from <internal:gem_prelude>:2:in `<internal:gem_prelude>'
D:/opt/fluent/lib/ruby/3.2.0/win32/registry.rb:910:in `encode': U+767E to ASCII-8BIT in conversion from UTF-16LE to UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError)
        from D:/opt/fluent/lib/ruby/3.2.0/win32/registry.rb:910:in `export_string'
        from D:/opt/fluent/lib/ruby/3.2.0/win32/registry.rb:611:in `each_key'
        from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:71:in `block (2 levels) in iterate_msys_paths'
        from D:/opt/fluent/lib/ruby/3.2.0/win32/registry.rb:435:in `open'
        from D:/opt/fluent/lib/ruby/3.2.0/win32/registry.rb:542:in `open'
        from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:70:in `block in iterate_msys_paths'
        from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:68:in `each'
        from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:68:in `iterate_msys_paths'
        from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:102:in `msys_path'
        from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:115:in `mingw_bin_path'
        from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/msys2_installation.rb:125:in `enable_dll_search_paths'
        from D:/opt/fluent/lib/ruby/site_ruby/3.2.0/ruby_installer/runtime/singleton.rb:27:in `enable_dll_search_paths'
        from D:/opt/fluent/lib/ruby/3.2.0/rubygems/defaults/operating_system.rb:24:in `<top (required)>'
        from D:/opt/fluent/lib/ruby/3.2.0/rubygems.rb:1332:in `require'
        from D:/opt/fluent/lib/ruby/3.2.0/rubygems.rb:1332:in `<top (required)>'
        from <internal:gem_prelude>:2:in `require'
        from <internal:gem_prelude>:2:in `<internal:gem_prelude>'
2024-02-28 14:24:25 +0800 [error]: Worker 0 exited unexpectedly with status 1
D:/opt/fluent/lib/ruby/3.2.0/rubygems/specification.rb:10:in `require_relative': Interrupt
        from D:/opt/fluent/lib/ruby/3.2.0/rubygems/specification.rb:10:in `<top (required)>'
        from D:/opt/fluent/lib/ruby/3.2.0/rubygems.rb:1325:in `require_relative'
        from D:/opt/fluent/lib/ruby/3.2.0/rubygems.rb:1325:in `<top (required)>'
        from <internal:gem_prelude>:2:in `require'
        from <internal:gem_prelude>:2:in `<internal:gem_prelude>'

Additional context
No response

It's known issue.
Please see https://github.com/fluent/fluent-package-builder/issues/616#issuecomment-1940324078 for the work around.


can't run daemon.js on 8gb home (starting server)

It's probably not been a requirement here, but I would like to run the script with a brand new save.  I can probably run some sub-scripts until i get enough to upgrade ram, but for OCD reasons I would like it to run from the start, on it's own.
Below is the output...
[home ~/]> free
Total:     8.00GB
Used:      0.00GB (0.00%)
Available: 8.00GB

[home ~/]> run daemon.js
Running script with 1 thread(s), pid 3 and args: [].
daemon.js: FAIL: (0 retries remaining): Run command returned no pid. Destination: /Temp/owned-source-files.txt.js Command: let result = ""; try { result = JSON.stringify(Object.fromEntries(ns.getOwnedSourceFiles().map(sf => [sf.n, sf.lvl]))); } catch { }
        if (ns.read("/Temp/owned-source-files.txt") != result) await ns.write("/Temp/owned-source-files.txt", result, 'w')
Ensure you have sufficient free RAM to run this temporary script.

I'm trying to understand what's happening to take the RAM.
I can see the file...
nano /Temp/owned-source-files.txt.js
import { formatMoney, formatNumberShort, formatDuration, parseShortNumber, scanAllServers } from "helpers.js"
export async function main(ns) { try { let result = ""; try { result = JSON.stringify(Object.fromEntries(ns.getOwnedSourceFiles().map(sf => [sf.n, sf.lvl]))); } catch { }
        if (ns.read("/Temp/owned-source-files.txt") != result) await ns.write("/Temp/owned-source-files.txt", result, 'w'); } catch(err) { ns.tprint(String(err)); throw(err); } }

From this I understand that ns.getOwnedSourceFiles() has a 5GB RAM cost.
I can see that daemon.js has a 5.5GB RAM cost, by doing this:
[home ~/]> run daemon.js --tail
Running script with 1 thread(s), pid 5 and args: [].
[home ~/]> free
Total:     8.00GB
Used:      5.50GB (68.75%)
Available: 2.50GB

-- Random Question -- Is there a better way to see the RAM cost of a script?
This means daemon.js would have to cost 3GB RAM or less in order to execute on the starter 8GB machine.
Has this been tried already?  Do you have any thoughts/advice?

random question answer...
[home ~/]> mem daemon.js
This script requires 5.50GB of RAM to run for 1 thread(s)

also,
[home ~/]> mem /Temp/owned-source-files.txt.js
This script requires 6.60GB of RAM to run for 1 thread(s)

I'll test it later, but I designed tryGetOwnedSourceFiles so that if it
fails (e.g. due to insufficient RAM) it'll raise an error message, but
default to assuming you have no owned source files, and everything should
continue working on that basis.
Is it actually crashing, or just raising lots of warnings/errors?
On Mon., Jan. 3, 2022, 1:06 a.m. cornernote, @.***>
wrote:

also,
[home ~/]> mem /Temp/owned-source-files.txt.js
This script requires 6.60GB of RAM to run for 1 thread(s)
—
Reply to this email directly, view it on GitHub
https://github.com/alainbryden/bitburner-scripts/issues/22#issuecomment-1003873211,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AARN33NLISO3KMNZBRNDJS3UUEVD5ANCNFSM5LEP4STA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

In fact, I was missing a try-catch to recover from the error after 5 failed attempts, so now it should still give warnings / an error indicating that it couldn't run, but it should suppress the actual exception and allow all programs to continue to run normally under the assumption that there are no owned source files.
It may keep happening periodically until RAM is expanded. If it's too much, I can handle the excessive noise separately in the daemon by caching the value once it's been returned once (if it's not already cached - I think it might be)

tried again now... it gave some RAM toast messages, but daemon is running!  :)
[home ~/]> run daemon.js
Running script with 1 thread(s), pid 2 and args: [].
[home ~/]> ps
(PID - 2) daemon.js
(PID - 12) hacknet-upgrade-manager.js -c --max-payoff-time 1h

Ummm seems we can use more RAM than we have...
[home ~/]> free
Total:             8.00GB
Used:             11.60GB (145.00%)
Available: -3600000000.00
[home ~/]> top
Script                                  PID       Threads         RAM Usage
daemon.js                               2         1               5.50GB
hacknet-upgrade-manager.js              12        1               6.10GB

Also, this message is the main noise.  Can it be run only if we have >8GB ?
daemon.js: FAIL: (0 retries remaining): Run command returned no pid. Destination: /Temp/sell-hashes-for-money.js Command: 0; if(ns.hacknet.spendHashes("Sell for Money")) ns.toast('Sold 4 hashes for $1M', 'success')
Ensure you have sufficient free RAM to run this temporary script.

I can tidy that up, it should only run if hacknet servers are available
On Mon., Jan. 3, 2022, 2:51 p.m. cornernote, @.***>
wrote:

Ummm seems we can use more RAM than we have...
[home ~/]> free
Total:             8.00GB
Used:             11.60GB (145.00%)
Available: -3600000000.00
[home ~/]> top
Script                                  PID       Threads         RAM Usage
daemon.js                               2         1               5.50GB
hacknet-upgrade-manager.js              12        1               6.10GB
Also, this message is the main noise. Can it be run only if we have >8GB ?
daemon.js: FAIL: (0 retries remaining): Run command returned no pid. Destination: /Temp/sell-hashes-for-money.js Command: 0; if(ns.hacknet.spendHashes("Sell for Money")) ns.toast('Sold 4 hashes for $1M', 'success')
Ensure you have sufficient free RAM to run this temporary script.
—
Reply to this email directly, view it on GitHub
https://github.com/alainbryden/bitburner-scripts/issues/22#issuecomment-1004287018,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AARN33ILLYUI4ABCR4DZECLUUHV3DANCNFSM5LEP4STA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you modified the open/close state.Message
ID: @.***>

Just wanted you to know that my latest changes make an effort to postpone a few scripts from starting up until there's sufficient RAM. 8GB is probably still too little.
I also fixed the hacknet issue mentioned in the comments.

I'd like to thank you for sharing your code.  I have learned so much from it!
After taking inspiration from your code, I wrote something that runs in 8GB... it can root servers, hack for profit, and manage hacknet!  It uses your concept of ns.write(random.js) to run pretty much every paid function in the background.
Have a play if you'd like to...
bitburner online
wget https://raw.githubusercontent.com/cornernote/bitburner-scripts/main/src/install.js install.js
run install.js
work

The key is this class -
https://github.com/cornernote/bitburner-scripts/blob/main/src/lib/Runner.js
It lets you write code like this...
let server = await runner.nsProxy['getServer']('n00dles')

See a really simple example here:
https://github.com/cornernote/bitburner-scripts/blob/main/src/samples/get-player-data.js

Looks pretty awesome. Good luck and have fun!
On Tue., Jan. 11, 2022, 10:52 p.m. cornernote, @.***>
wrote:

I'd like to thank you for sharing your code. I have learned so much from
it!
After taking inspiration from your code, I wrote something that runs in
8GB... it can root servers, hack for profit, and manage hacknet! It uses
your concept of ns.write(random.js) to run pretty much every paid function
in the background.
Have a play if you'd like to...
bitburner online https://danielyxie.github.io/bitburner/
wget https://raw.githubusercontent.com/cornernote/bitburner-scripts/main/src/install.js install.js
run install.js
work
The key is this class -
https://github.com/cornernote/bitburner-scripts/blob/main/src/lib/Runner.js
It lets you write code like this...
let server = await runner.nsProxy'getServer'
See a really simple example here:
https://github.com/cornernote/bitburner-scripts/blob/main/src/samples/get-player-data.js
—
Reply to this email directly, view it on GitHub
https://github.com/alainbryden/bitburner-scripts/issues/22#issuecomment-1010578696,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AARN33IXJ46UEBUEFKUJDQDUVTUGBANCNFSM5LEP4STA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you modified the open/close state.Message
ID: @.***>


[dcv:plot] Invalid imshow on OS X/macOS

Images are shown in 3rd quadrant of a window:

imread/imwrite performs correctly, so I assume bug must be somewhere in gl/glfw.

Seems to be related to glfw/glfw#49


4.9.3

the test failure is
FAIL: upgrade_test.go:9: QemuSuite.TestUpgrade
1912s
5872
upgrade_test.go:22:
1912s
5873
    s.Reboot(c)
1912s
5874
common_test.go:141:
1912s
5875
    c.Assert(s.WaitForSSH(), IsNil)
1912s
5876
... value *errors.errorString = &errors.errorString{s:"Failed to connect to SSH: exit status 255"} ("Failed to connect to SSH: exit status 255")

not really helping :(
I'm going to merge, because its not related to this PR


Add support for Ubuntu 24.04

Please add a Debian-24.yml file under /vars with the same packages that are in Debian-22.yml to add support for Ubuntu 24.04.  I have tested this, and it works.  I'd open up a PR myself, but don't have access.  Thanks.

Never mind, figured out how to submit a pull request from forked repo.  PR is here: https://github.com/mahdi22/linux_joindomain/pull/9

Hello,
Thank you for your participation. Your requedt is merged