Upload server is timing out

Description
Please describe the issue you are facing using the CLI.
Connections to upload.appcenter.ms are timing out.  Tried from multiple geographic locations, same result.
Repro Steps
Please list the steps used to reproduce your issue.
nc -v upload.appcenter.ms 443
nc: connectx to upload.appcenter.ms port 443 (tcp) failed: Operation timed out

dig upload.appcenter.ms

; <<>> DiG 9.10.6 <<>> upload.appcenter.ms
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7596
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;upload.appcenter.ms.
Details

Is there a particular beacon/command that you are having trouble with?

e.g. appcenter tokens list

Which version of the App Center CLI are you using?

e.g. 2.7.4

Which OS version did you experience the issue on?

e.g. Windows 10 version 20H2, Ubuntu 20.04.1 LTS, macOS BigSur 11.1

What command line/shell are you using?

e.g. Shell 5.8 /bin/zsh

What version of Node.js and NPM/Yarn are you using?

e.g. node v12.19.0, npm 6.14.8

Additionally, you can provide verbose logs of a CLI command by using DEBUG="appcenter-cli:*" before the command, for example, DEBUG="appcenter-cli:*" appcenter apps list. Include the logs here, don't forget to omit any personal/sensitive data.

I noticed that the status page declares that the host upload.appcenter.ms is "discontinued", but when a request is made against the API:
curl -X POST --header 'Content-type: application/json' --header 'Accept: application/json' --header 'X-API-Token: REDACTED' https://api.appcenter.ms/v0.1/apps/REDACTED/REDACTED/release_uploads

the upload_url specifies:
"upload_url":"https://upload.appcenter.ms/v0.1/apps/...

Do you use the appcenter-cli for uploading new releases or do you try to upload new releases via API manually?
In the second case you probably use the deprecate endpoint and you need to switch to this. It doesn't look like the appcenter-cli issue. If after updating API this problem still persist so please open a support ticket on the App Center portal.

In the second case you probably use the deprecate endpoint and you need to switch to this.

That documentation is - at best - inaccurate.
Here is a tested and working example in bash:
APPCENTER_OWNER_NAME=${APPCENTER_OWNER_NAME:-"your-owner-name"}
APPCENTER_APP_NAME=${APPCENTER_APP_NAME:-"your-app-name"}

log DEBUG "APPCENTER_OWNER_NAME=${APPCENTER_OWNER_NAME}"
log DEBUG "APPCENTER_APP_NAME=${APPCENTER_APP_NAME}"
log DEBUG "APPCENTER_DISTRIBUTION_GROUP_ID=${APPCENTER_DISTRIBUTION_GROUP_ID}"
log DEBUG "APPCENTER_NOTIFY_TESTERS=${APPCENTER_NOTIFY_TESTERS}"

log INFO "Upload the Android APK to ${APPCENTER_OWNER_NAME}/${APPCENTER_APP_NAME}"
log DEBUG "Fetching upload parameters"
#    Create an upload resource and get an upload_url (good for 24 hours): The endpoint to call is
#    POST /v0.1/apps/{owner_name}/{app_name}/uploads/releases.
APPCENTER_RELEASE_UPLOAD=$(
    curl "https://api.appcenter.ms/v0.1/apps/${APPCENTER_OWNER_NAME}/${APPCENTER_APP_NAME}/uploads/releases" \
        -s --fail --retry 5 \
        -X POST \
        --header 'Content-Type: application/json' \
        --header 'Accept: application/json' \
        --header "X-API-Token: ${APPCENTER_API_TOKEN}" \
        --data "{
            \"release_id\": ${CI_PIPELINE_ID},
            \"build_version\": \"${VERSION}\",
            \"build_number\": \"${CI_PIPELINE_ID}\"
        }"
)
#    The result will look something like this, with {VARIABLE_NAME} replacing data unique to each use:
#
#    {
#        "id": "{RELEASES_ID}",
#        "package_asset_id": "{APPCENTER_PACKAGE_ASSET_ID}",
#        "upload_domain": "{APPCENTER_UPLOAD_DOMAIN}",
#        "token": "{TOKEN}",
#        "url_encoded_token": "{APPCENTER_URL_ENCODED_TOKEN}"
#    }
APPCENTER_RELEASE_ID=$(echo ${APPCENTER_RELEASE_UPLOAD} | jq -r '.id')
APPCENTER_PACKAGE_ASSET_ID=$(echo ${APPCENTER_RELEASE_UPLOAD} | jq -r '.package_asset_id')
APPCENTER_UPLOAD_DOMAIN=$(echo ${APPCENTER_RELEASE_UPLOAD} | jq -r '.upload_domain')
APPCENTER_URL_ENCODED_TOKEN=$(echo ${APPCENTER_RELEASE_UPLOAD} | jq -r '.url_encoded_token')
APK_FILE_NAME="${CI_PROJECT_NAME}-app-${VERSION}-${CI_ENVIRONMENT_NAME}.apk"
APK_FILE_SIZE_BYTES=$(wc -c ${APK_FILE_NAME} | awk '{print $1}')

log DEBUG "APPCENTER_RELEASE_ID=${APPCENTER_RELEASE_ID}"
log DEBUG "APPCENTER_PACKAGE_ASSET_ID=${APPCENTER_PACKAGE_ASSET_ID}"
log DEBUG "APPCENTER_UPLOAD_DOMAIN=${APPCENTER_UPLOAD_DOMAIN}"
log DEBUG "APPCENTER_URL_ENCODED_TOKEN=${APPCENTER_URL_ENCODED_TOKEN}"
log DEBUG "APK_FILE_NAME=${APK_FILE_NAME}"
log DEBUG "APK_FILE_SIZE_BYTES=${APK_FILE_SIZE_BYTES}"

log DEBUG "Setting upload metadata"
APPCENTER_UPLOAD_METADATA=$(
    curl -s "${APPCENTER_UPLOAD_DOMAIN}/upload/set_metadata/${APPCENTER_PACKAGE_ASSET_ID}?file_name=${APK_FILE_NAME}&file_size=${APK_FILE_SIZE_BYTES}&token=${APPCENTER_URL_ENCODED_TOKEN}&content_type=application/vnd.android.package-archive" \
        -d POST \
        --header 'Content-Type: application/json' \
        --header 'Accept: application/json' \
        --header "X-API-Token: ${APPCENTER_API_TOKEN}"
)
#    The output returned should look something like this:
#
#    {
#        "error":false,
#        "id":"{APPCENTER_UPLOAD_ID}",
#        "chunk_size":4194304,
#        "resume_restart":false,
#        "chunk_list":[1,2,3,4,5],
#        "blob_partitions":1,
#        "status_code":"Success"
#    }
APPCENTER_UPLOAD_ID=$(echo ${APPCENTER_UPLOAD_METADATA} | jq -r '.id')
APPCENTER_CHUNK_SIZE=$(echo ${APPCENTER_UPLOAD_METADATA} | jq -r '.chunk_size')
APPCENTER_CHUNK_DIR=$(mktemp -d)

log DEBUG "APPCENTER_UPLOAD_ID=${APPCENTER_UPLOAD_ID}"
log DEBUG "APPCENTER_CHUNK_SIZE=${APPCENTER_CHUNK_SIZE}"
log DEBUG "APPCENTER_CHUNK_DIR=${APPCENTER_CHUNK_DIR}"

log DEBUG "Split the APK in chunks"
#    Using the chunk_size value, you can split your app upload into sequential chunks for upload to Distribute.
#    This command generates sequential files in the APPCENTER_CHUNK_DIR directory named splitaa, splitab, and so on.
#    Each file is split within the chunk_size limit.
split -b ${APPCENTER_CHUNK_SIZE} ${APK_FILE_NAME} ${APPCENTER_CHUNK_DIR}/split

log DEBUG "Uploading APK chunks"
#    Next you need to upload each chunk of the split app package with the respective block:
BLOCK_NUMBER=0
for i in ${APPCENTER_CHUNK_DIR}/*; do
    BLOCK_NUMBER=$(($BLOCK_NUMBER + 1))
    CONTENT_LENGTH=$(wc -c "$i" | awk '{print $1}')

    curl "${APPCENTER_UPLOAD_DOMAIN}/upload/upload_chunk/${APPCENTER_PACKAGE_ASSET_ID}?token=${APPCENTER_URL_ENCODED_TOKEN}&block_number=${BLOCK_NUMBER}" \
    --fail --retry 5 \
        -X POST \
        --data-binary "@$i" \
        --header 'Content-Type: application/vnd.android.package-archive' \
        --header "Content-Length: ${CONTENT_LENGTH}"
done

log DEBUG "Finishing the APK upload"
#    After the upload is done, update the upload resource's status to uploadFinished.
curl "${APPCENTER_UPLOAD_DOMAIN}/upload/finished/${APPCENTER_PACKAGE_ASSET_ID}?token=${APPCENTER_URL_ENCODED_TOKEN}" \
    -d POST \
    --header 'Content-Type: application/json' \
    --header 'Accept: application/json' \
    --header "X-API-Token: ${APPCENTER_API_TOKEN}"

log DEBUG "Committing the APK upload"
curl "https://api.appcenter.ms/v0.1/apps/${APPCENTER_OWNER_NAME}/${APPCENTER_APP_NAME}/uploads/releases/${APPCENTER_RELEASE_ID}" \
    --fail --retry 5 \
    -X PATCH \
    --header 'Content-Type: application/json' \
    --header 'Accept: application/json' \
    --header "X-API-Token: ${APPCENTER_API_TOKEN}" \
    --data '{"upload_status": "uploadFinished"}'

log DEBUG "Waiting for the APK upload confirmation"
#    Once uploaded, there is a short delay before the upload is marked as finished.
#    Poll for this status to get the $APPCENTER_DISTINCT_RELEASE_ID for the next step:
APPCENTER_DISTINCT_RELEASE_ID='null'
for i in {1..10}; do
    APPCENTER_POLL_RESULT=$(
        curl -s "https://api.appcenter.ms/v0.1/apps/${APPCENTER_OWNER_NAME}/${APPCENTER_APP_NAME}/uploads/releases/${APPCENTER_RELEASE_ID}" \
        --header 'Content-Type: application/json' \
        --header 'Accept: application/json' \
        --header "X-API-Token: ${APPCENTER_API_TOKEN}"
)
    APPCENTER_DISTINCT_RELEASE_ID=$(echo ${APPCENTER_POLL_RESULT} | jq -r '.release_distinct_id')

    if [[ ${APPCENTER_DISTINCT_RELEASE_ID} == 'null' ]]; then
        log WARN "Failed to find release on AppCenter, retrying in 10 seconds..."
        sleep 10;
    else
        break
    fi
done

if [[ ${APPCENTER_DISTINCT_RELEASE_ID} == 'null' ]]; then
    log ERROR "Failed to find release on AppCenter"
    exit ${ERROR_APPCENTER_RELEASE_NOT_FOUND}
fi

log DEBUG "APPCENTER_DISTINCT_RELEASE_ID=${APPCENTER_DISTINCT_RELEASE_ID}"

log DEBUG "Setting release notes"
APPCENTER_RELEASE_NOTE=$(
    curl "https://api.appcenter.ms/v0.1/apps/${APPCENTER_OWNER_NAME}/${APPCENTER_APP_NAME}/releases/${APPCENTER_DISTINCT_RELEASE_ID}" \
        --fail --retry 5 \
        -X PATCH \
        --header "X-API-Token: ${APPCENTER_API_TOKEN}" \
        --header 'Content-Type: application/json' \
        --data "{
            \"release_notes\": \"Version ${VERSION} (${CI_ENVIRONMENT_NAME})\"
        }"
)

log DEBUG "APPCENTER_RELEASE_NOTE=${APPCENTER_RELEASE_NOTE}"

if [ -z "${APPCENTER_DISTRIBUTION_GROUP_ID}" ]; then
    log WARN "APPCENTER_DISTRIBUTION_GROUP_ID is not set: the Android APK release will not be distributed"
else
    log DEBUG "Distributing release to group"
    if [ "${APPCENTER_NOTIFY_TESTERS}" != 'false' ]; then
        APPCENTER_NOTIFY_TESTERS='true'
    else
        log WARN "APPCENTER_NOTIFY_TESTERS is not set to 'true': the distribution group users will not be notified"
    fi
    APPCENTER_DISTRIBUTION=$(
        curl "https://api.appcenter.ms/v0.1/apps/${APPCENTER_OWNER_NAME}/${APPCENTER_APP_NAME}/releases/${APPCENTER_DISTINCT_RELEASE_ID}/groups" \
            --fail --retry 5 \
            -X POST \
            --header "X-API-Token: ${APPCENTER_API_TOKEN}" \
            --header 'Content-Type: application/json' \
            --data "{
                \"id\": \"${APPCENTER_DISTRIBUTION_GROUP_ID}\",
                \"mandatory_update\": true,
                \"notify_testers\": ${APPCENTER_NOTIFY_TESTERS}
            }"
    )
    log DEBUG "APPCENTER_DISTRIBUTION=${APPCENTER_DISTRIBUTION}"
fi

log INFO "Success: the Android app for the ${CI_ENVIRONMENT_NAME} environment has been published!"

Hi, @JMLX42 ! Thanks for the details!
Looks like it's not an appcenter-cli issue, so I'm going to close it. Please open a support ticket on the App Center portal.


dashd crashes after updating pruned node from 0.13 to <IP_ADDRESS>

Describe the issue
I tried to upgrade a pruned node with prune=1024 from version <IP_ADDRESS> to version <IP_ADDRESS>.
dashd crashes during startup with the following message:
dashd: llmq/quorums_blockprocessor.cpp:276: void llmq::CQuorumBlockProcessor::UpgradeDB(): Assertion 'r' failed.
The reason seems to be that DIP0003EnforcementHeight is less than pruneheight and therefore llmq::CQuorumBlockProcessor::UpgradeDB() tries to access a pruned block.
Can you reliably reproduce the issue?
yes
What version of Dash Core are you using?
v<IP_ADDRESS>

resolved by 2962


is_module_installed fails if temp doesn't have write permissions

Description
What steps will reproduce the problem?

What is the expected output? What do you see instead?
Please provide any additional information below
File "/home/rogelio/dev/anaconda/lib/python3.6/site-packages/spyder/plugins/ipythonconsole.py", line 873, in run_script
self.create_client_for_file(filename)
File "/home/rogelio/dev/anaconda/lib/python3.6/site-packages/spyder/plugins/ipythonconsole.py", line 1363, in create_client_for_file
self.create_new_client(filename=filename)
File "/home/rogelio/dev/anaconda/lib/python3.6/site-packages/spyder/plugins/ipythonconsole.py", line 994, in create_new_client
interpreter=pyexec)
File "/home/rogelio/dev/anaconda/lib/python3.6/site-packages/spyder/utils/programs.py", line 405, in is_module_installed
fd, script = tempfile.mkstemp(suffix='.py', dir=TEMPDIR)
File "/home/rogelio/dev/anaconda/lib/python3.6/tempfile.py", line 342, in mkstemp
return _mkstemp_inner(dir, prefix, suffix, flags, output_type)
File "/home/rogelio/dev/anaconda/lib/python3.6/tempfile.py", line 260, in _mkstemp_inner
fd = _os.open(file, flags, 0o600)
FileNotFoundError: [Errno 2] No existe el archivo o el directorio: '/tmp/spyder-rogelio/tmpoajeqzou.py'
Version and main components

Spyder Version: 3.2.4
Python Version: 3.6.3
Qt Versions: 5.6.2, PyQt5 5.6 on Linux

Dependencies
pyflakes >=0.6.0 :  1.6.0 (OK)
pycodestyle >=2.3:  2.3.1 (OK)
pygments >=2.0   :  2.2.0 (OK)
pandas >=0.13.1  :  0.20.3 (OK)
numpy >=1.7      :  1.13.3 (OK)
sphinx >=0.6.6   :  1.6.3 (OK)
rope >=0.9.4     :  0.10.5 (OK)
jedi >=0.9.0     :  0.10.2 (OK)
psutil >=0.3     :  5.4.0 (OK)
nbconvert >=4.0  :  5.3.1 (OK)
sympy >=0.7.3    :  1.1.1 (OK)
cython >=0.21    :  0.26.1 (OK)
qtconsole >=4.2.0:  4.3.1 (OK)
IPython >=4.0    :  6.1.0 (OK)
pylint >=0.25    :  1.7.4 (OK)

It seems you don't have write permissions in your /tmp directory. Spyder needs to write to it to run some validations in the Python interpreter you selected.

@ccordoba12 Does this need to be fixed on the Spyder side, or can it be closed at this point?

Let's close it because of lack of response. Without write permissions on /tmp, Spyder can't work.


--timedate parameter addition

The new --timedate parameter is a way that users can specify a date and time they would like to download/extract Starchive containers from.  This gives them much more control and can significantly speed up the process. Since they no longer have to download the entire Starchive.
For instance, if the node was offline or out of consensus for 7 days... they could specify that date and Starchiver will look at the hash file to determine which container would be a good starting point to ensure they only download/extract what they need.
Their are multiple ways --timedate can be used...
It can be used with a unix style timestamp, also this UTC format "YYYY-MM-DD zHMM" or "YYYY-MM-DD zHMM", it can be this format "YYYY-MM-DD.HH", or if no data is given... it will automatically search for the most recent snapshot timestamp.
Then it will use this data to determine with Starchive container to start at.

In this version it purposely goes back 1 hour from the timestamp. But it's not perfect since the mtime can be very different on the users server. So... In this case if a user has an issue they can send a specific date and time to go back to.
In the next version I am going to make sure the mtime of all snapshots compiled inside the Starchiver Containers are matching the timestamp of the snapshot on the API. Then I will change this code to always look at the timestamp on the API instead of the mtime of the file itself.
It will definitely get more testing once it's released though. But I'm pretty confident it will work. :-)


Crashes when parsing an array as a return value

When parsing a contract that has a function which returns an array Solium chrashes.
To reproduce, lint this:
contract Test {
    function crash() returns (address[]) {
    }
}

@oed are you sure that address[] is a valid usage?
I looked up in docs (http://solidity.readthedocs.io/en/develop/types.html), my interpretation is that it currently doesn't support dynamic arrays. Solc compiles the code but also produces a formal error that this is not supported.
The doc says:
Due to limitations of the EVM, it is not possible to return dynamic content from external function calls. The function f in contract C { function f() returns (uint[]) { ... } } will return something if called from web3.js, but not if called from Solidity.

The only workaround for now is to use large statically-sized arrays.

@duaraghav8 I think it is supported, as long as you don't use the function with an internal call. External calls are fine.
http://solidity.readthedocs.io/en/latest/frequently-asked-questions.html?highlight=return array#can-you-return-an-array-or-a-string-from-a-solidity-function-call
Returning variably-sized data as part of an external transaction or call is fine.

@oed I'm working on a fix for this :)

@oed v0.5.1 resolves this, please re-open the issue if you still face any trouble with this.


Refactor Affine

This patch refactors affine to make the code more
readable and change the matrix generation routine to
a numpy-based one. It also merges the matrix
generation of Affine and pillike.Affine and lays
the foundation for adding a center parameter to
Affine.
This patch also changes the shear mechanic in
Affine. When shearing on the x-axis, the points at
the top are now only moved to the left/right and no
longer up/down. Previously, they were also slightly
moved up/down. (Analogous for the y-axis.)

Codecov Report

Merging #639 into master will decrease coverage by 1.35%.
The diff coverage is 99.00%.

@@            Coverage Diff             @@
##           master     #639      +/-   ##
==========================================
- Coverage   96.37%   95.02%   -1.35%
==========================================
  Files          41       41
  Lines       14683    14729      +46
==========================================
- Hits        14150    13995     -155
- Misses        533      734     +201


Variables: Improve styling and spacing

Before (not really true before as this is with

After:

@dprokop it's a cool idea, but variables can also have specific display name (that cannot be used in query), and from a consumer perspective of the variable the fact that these are used in queries should not be that important, just just a filter


Move pull request template location so that it works by default

Fixes #13.
Proposed Changes

Move pull request template location to somewhere else so that hopefully it will actually work :)

Sweet, it works!


Make package pip-installable

did this so you can install the package via the pip install git+... method - not sure it's 100% necessary but this works reasonably well.


why does opengl load both the binding and opengl32

Question
does the native bindings need opengl32.dil I can’t find a direct reference but it loads both that confused me I thought it was a bug opengl32 was loaded
Thanks spasi you done a good job by the way
I only know basic java and no native code

If you build LWJGL locally and run ant demo -Dclass=org.lwjgl.demo.glfw.Gears -Djvmargs="-Dorg.lwjgl.util.Debug=true -Dorg.lwjgl.util.DebugLoader=true", you'll see the following:
[LWJGL] Loading JNI library: lwjgl_opengl
        Module: org.lwjgl.opengl
        Loaded from classpath: D:\PROJECTS\lwjgl3\bin\libs\native\windows\x64\org\lwjgl\opengl\lwjgl_opengl.dll
[LWJGL] Loading library: opengl32
        Module: org.lwjgl.opengl
        Loaded from system paths: C:\Windows\SYSTEM32\opengl32.dll

Both libraries are necessary for the OpenGL bindings to work.
The second library is straightforward: it's the OpenGL library provided by the operating system, the one actually exposing the OpenGL API. You can think of it as your GPU's OpenGL driver (there is some indirection going on, but you can ignore all that).
The first library is a typical JNI library. It provides the implementation for every native method inside the OpenGL module. For example, for this Java method:
public static native long nglGetString(int name);

@Nullable
@NativeType("GLubyte const *")
public static String glGetString(@NativeType("GLenum") int name) {
    long __result = nglGetString(name);
    return memUTF8Safe(__result);
}

the native code for nglGetString lives in the lwjgl_opengl native library:
JNIEXPORT jlong JNICALL Java_org_lwjgl_opengl_GL11C_nglGetString(JNIEnv *__env, jclass clazz, jint name) {
    glGetStringPROC glGetString = (glGetStringPROC)tlsGetFunction(111);
    UNUSED_PARAM(clazz)
    return (jlong)glGetString(name);
}

Most LWJGL bindings do not need a second native library to work. OpenGL and OpenGL ES are special, because OpenGL contexts (and their corresponding function pointers) are thread-local. Different threads may have to call different functions. In order to expose a static API (OpenGL code in Java that looks the same as the corresponding C code), a thread-local lookup has to happen somewhere. This is what the tlsGetFunction does. The details are complicated (see ThreadLocalUtil if interested), but this particular implementation is more efficient than doing the thread-local lookup in Java.
LWJGL 4 will migrate to Project Panama and since that's a huge API breaking change, it will be a good opportunity to refactor the OpenGL bindings with a non-static API, similar to JOGL. For example:
// instead of
private String lwjgl3() {
    return glGetString(GL_VERSION);
}

// you'd write
private String lwjgl4(GL gl) {
    return gl.GetString(GL_VERSION);
}

It has the drawback that you'll have to pass the OpenGL context around, but it eliminates all thread-local issues.

Thanks thought it was something like that the bindings needed opengl32


Cannot Declare UDF

Trying to declare a newly UDF written by me:
declare external function F_SET_INTERSECT
cstring(255), cstring(255), cstring(1)
returns integer by value
entry_point 'SET_INTERSECT'
module_name 'GP_UDF';
Flamerobin hangs indefinitely on "Starting Transaction", previous Flamerobin  Version (January?) had same problem.
Declaring the UDF with iSql works succesfully, andthen the  UDF itself work as expected,
FlameRobin 0.9.3 (git hash 78287cd4) Unicode, Win32
Server Version: WI-V<IP_ADDRESS>374 Firebird 3.0, 64 bit

Confirm it's hangs here too in the latest source code
Both 32 and 64 bits

Ok with <IP_ADDRESS>


search_tweets.py - Remove tweet_parser import [v2]

After running pip install searchtweets-v2 and manually installing python-dateutil, running search_tweets.py -h throws the following exception:
Traceback (most recent call last):
  File "/home/jordan/Development/test/venv/bin/search_tweets.py", line 10, in <module>
    from searchtweets import (ResultStream,
  File "/home/jordan/Development/test/venv/lib/python3.6/site-packages/searchtweets/__init__.py", line 4, in <module>
    from .result_stream import ResultStream, collect_results
  File "/home/jordan/Development/test/venv/lib/python3.6/site-packages/searchtweets/result_stream.py", line 19, in <module>
    from tweet_parser.tweet import Tweet
ModuleNotFoundError: No module named 'tweet_parser'

The documentation mentions tweet_parser isn't used in v2, so I removed the import in result_stream.py and was able to fetch tweets using search_tweets.py
Can this import be removed in v2?

I can create the PR to remove the import if you think it's acceptable.

Hey there, while I could not reproduce any issues with search_tweets.py -h, I went ahead and removed any references to the TweetParser package. Had been on my list for a while, but an earlier attempt got tripped up.
I just updated the repository and rerolled the pypi package.
Thanks!


random tinkering

Some build findings for 3box:

because of lots of Node.js C bindings the dependency install time went up like 1-2 min.
the build time went up by 3 min., until the error
package introduces 71 security vulnerabilities marked as high

As for build error, looks like we need to prevent stuff in the Profile provider from happening when we are not in the browser. All providers are currently wrapped around browser and SSR build, so for SSR build we can leave the provider, but then need to add e.g. typeof window !== 'undefined' before initializing things in there, so it only fires in browser

Yes, the lib is pretty big, we can optimize using https://github.com/3box/3box-js#build  and lazy loading the full library only when the user wants to update the profile.

and main production app.js delivered first to user went up by 2MB, definitely needs to be lazy loaded.

Note: this PR seems to be related to Market-240

When you click the 3box link ideally it should open in a new tab

lgtm

Tested it with https://www.3box.io/0x655efe6eb2021b8cefe22794d90293aec37bb325
Works!

Works wonderful.

have one small remaining issue on small viewports, but it's no blocker. Somehow tippy.js creates a body overflow although the popover fits perfectly within the viewport. This leads to some weird interaction where the body behind the popover scrolls a bit while the popover stays correctly placed. None of our css is causing that and it seems an issue with tippy.js


Fix product listing when 'Hide products after clearance' is active

1. Why is this change necessary?
It will fix the product listings when Hide products after clearance and elastic search is active.
2. What does this change do, exactly?
It inverts the filter for isCloseout manually since it seems this is nothing elastic search can do by itself.
3. Describe each step to reproduce the issue or behaviour.
Activate Hide products after clearance in the administration.
4. Please link to the relevant issues (if any).
https://issues.shopware.com/issues/NEXT-10003
5. Checklist

[ ] I have written tests and verified that they fail without my change
[x] I have squashed any insignificant commits
[ ] I have created a changelog file with all necessary information about my changes
[ ] I have written or adjusted the documentation according to my changes
[ ] This change has comments for package types, values, functions, and non-obvious lines of code
[x] I have read the contribution requirements and fulfil them.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hey @ddauenheimer , thank you for creating this pull request!
Please add some tests for your change and sign the CLA, I'll be happy to import this then.

Hey @ddauenheimer, we closed this PR due to inactivity. Should you still be interested in your changes being merged please resolve the remaining issues and create a new PR.


fix: client generator

Fixes https://github.com/the-codegen-project/cli/issues/149
Fixes https://github.com/the-codegen-project/cli/issues/148
Additional fix:

Generators can now depend on generators that also depend on other generators

:tada: This PR is included in version 0.22.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


HelmInstaller@1 Unexpected HTTP response: 403

For the last couple of days, our HelmInstaller task keeps giving the following error and then installs an older version of helm. Retrying failed jobs usually fixes. Can you not add a fallback version parameter so that we can at least specify a better version to use than 3.1.2 as the subsequent stage fails due to this old version?
##[warning]Cannot get the latest Helm info from https://api.github.com/repos/helm/helm/releases. Error Error: Unexpected HTTP response: 403. Using default Helm version v3.1.2.

Question, Bug, or Feature?
Type: Bug
Enter Task Name: HelmInstaller@1
Environment

Server - Azure Pipelines

account name: Kelvion-DevOps
team project name: MyAFC
build definition name: myafc-helm (kts)
build number: 2022.12.16-2

Agent - Hosted

agent queue name: Azure Pipelines

Issue Description
From my point of view a failed download of a Helm release from GitHub is a transient error. The fallback behaviour of the pipeline task is to install an outdated Helm version (3.1.2 - released in march 2020) that does not support common options/flags used as of today - and that breaks the build/deployment in subsequent pipeline tasks using HelmDeploy@0 unless you explicitly verify the Helm version.
My suggestion for a fallback behaviour of HelmInstaller@1 : use a recent Helm version if installed in the hosted build agent image. An example: hosted agents with image ubuntu-22.04 ship with Helm 3.10.2 - see Ubuntu 22.04 runner image (package management)

Yes I figured it was a transient issue but 3 days is a pain. Good suggestion though, forgot that it was probably pre installed on the agent so can probably just remove the HelmInstaller@1 task. Be interesting to see how often the pre installed one gets updated though


fix: disable scroll y when sign in is on page

Pull Request Information
Issue Number: #114
Description of Changes:
I worked on a fix to disable vertical scroll on Sign in page
Screenshots/GIFs:
The scroll is disabled when the sign-in page is shown

It is re-enabled when home page is shown

Checklist
Please make sure that your PR meets the following requirements:

[x ] Code is well-documented.
[ x] Code follows the project's coding style and standards.
[ ] New code includes relevant unit tests (if applicable).
[ ] Existing tests have been updated (if applicable).
[ x] The PR has been reviewed for correctness and quality.
[x ] The PR is up-to-date with the latest changes from the main branch.
[x ] The PR does not introduce new issues or conflicts.

Additional Notes
N/A
Reviewers
Please tag any specific team members or contributors who should review this pull request. For general review, you can use @flavour-fusion/team.
Checklist for Reviewers

[x] The code changes are correct and functional.
[x] Code adheres to project coding style and standards.
[ ] Unit tests (if applicable) have been added or updated.
[ ] The code does not introduce new issues or conflicts.
[x] All relevant changes have been documented in the project's documentation (if applicable).

By submitting this pull request, I confirm that I have read and agree to the Contributor License Agreement, and I'm submitting this code in accordance with the terms of that agreement.

@c4coderandcreator looks like I cannot add reviewers to my pull request. Kindle give instructions on how to do it ! Thanks so much

@c4coderandcreator looks like I cannot add reviewers to my pull request. Kindle give instructions on how to do it ! Thanks so much

I am reviewing it !! It seems alright !!


Merge renovate under one PR

To avoid number of PRs tho

Closed in https://github.com/kodadot/nft-gallery/commit/a519f2591d76b7c809737c054dfd197762dfa674


Assistant selectively using a British Voice

Hi, I thought yourselves at Google in charge of the AIY repo would be interested to know that the AIY platform is now selectively using a British voice
If I ask what is the time - the response is an American voice
If I ask will it rain today - the response is a British voice
If I ask do I need an Umbrella today - the response is a British voice
If I ask what is the temperature - the response is a British voice
All of the British accent responses are followed by this error - [1557:1569:ERROR:assistant_output_handler.cc(543)] Ignoring kUnimplemented action error!`
I haven't done anything to update the assistant software so presumable this is a server side change.
Can you tell me the correct repo against which I should raise an issue????
(

assistant-sdk-python would be the right place for this. Thanks for reporting!

I raised the issue
https://github.com/googlesamples/assistant-sdk-python/issues/140#issuecomment-353140432
They asked my to try google-assistant-library 0.1.0 (we are currently using google-assistant-library-0.0.3)
I updated my env to  google-assistant-library 0.1.0 and then ran  src/assistant_library_with_local_commands_demo.py
It broke -
(env) pi@raspberrypi:~/AIY-voice-kit-python $ src/assistant_library_with_local_commands_demo.py
Traceback (most recent call last):
File "src/assistant_library_with_local_commands_demo.py", line 99, in
main()
File "src/assistant_library_with_local_commands_demo.py", line 93, in main
with Assistant(credentials) as assistant:
TypeError: init() missing 1 required positional argument: 'device_model_id'
Does something need to happen with the AIY software to make it compatible with google-assistant-library 0.1.0?


Add some statistics report in public

What's the problem this feature will solve?
It will help the developers to choose library wisely. More downloads are shown means more reliable and have better support than others.
Describe the solution you'd like
Add statistics of the following

Total Downloads
Weekly Downloads
Rating (depends on downloads and release and issue closed)

Additional context

NIL

I think this is a duplicate of #787 and pypa/pypi-legacy#254 (for the statistics), and #991 (for the rating, although this is not exactly what you were suggesting).

Closing as a duplicate of #787 and #991.


email messages to aliases being rejected

I would like to receive emails to my aliases addresses.
Context
I defined my aliases in
config/postfix-aliases.cf
with the syntax
alias1@domaain:     mymailbox@domain
alias2@domain:     mymailbox@domain

It does not work.
Expected Behavior
I want mail sent to alias1|2@domain will be read in my mailbox@domain together wil other messages sent directly to  mailbox@domain .
Actual Behavior
email rejected.
Possible Fix
no idea, everything done as documented (I think)
Steps to Reproduce

edit the config/postfix-aliases.cf
restart the container
send email to alias address
cannot deliver

Your Environment

Amount of RAM available: 4Gb
Mailserver version used: installed the container in January 2020
Docker version used: Docker version 19.03.6, build 369ce74a3c
Environment settings relevant to the config:cannot find one specific
Any relevant stack traces ("Full trace" preferred):

In logs:
Mar  2 08:55:51 vps postfix/lmtp[1585]: 1AB6D7017BF4<EMAIL_ADDRESS>relay=....[/var/run/dovecot/lmtp], delay=0.06, delays=0.02/0.02/0.02/0.01, dsn=5.1.1, status=bounced (host ...[/var/run/dovecot/lmtp] said: 550 5.1.1<EMAIL_ADDRESS>User doesn't exist<EMAIL_ADDRESS>(in reply to RCPT TO command))
(actual names hidden)

How can I debug aliases delivery?

I have to report also that I edited the postfix-aliases.cf after run of the setup.sh, which I hope is correct. Afterwards I did:
docker mail stop
docker mail start

Try to recreate the container instead of stopping and starting it:
docker-compose down
docker-compose up -d mail

The postfix-aliases.cf file should either be processed when the container is started the first time by the startup script or by another script that checks for changes. Re-creating the container will make sure that it is picked up.
If it still doesn't work, please check the logs to see if there are errors when the aliases are processed or if there are other errors or warnings later. You could also enable debug logging with DMS_DEBUG=1 to get even more logs.

What will happen to the users maildir if I use
docker-compose down

I would like not to loose users emails since they access through IMAP

apparently they are saved in the host folder /var/mail/<username> , correct?

done, no way same things. "User does not exists" in logs.

apparently they are saved in the host folder /var/mail/ , correct?

Depends on your setup. If you used docker-compose.yml.dist to get started, then emails are saved in docker volumes, which probably are located somewhere in /var/lib/docker/volumes/ on the host.

I edited the postfix-aliases.cf

Aliases are defined in config/postfix-virtual.cf.
Or just use: setup.sh alias add alias1@domain mymailbox@domain

no way, same problem, tried both ways.
Now in config/postfix-virtual.cf I have:
alias1@mydomain    myuser

but I try to send to alias1@mydomain I alwaws get "User doesn't exists"

Did you restart the container?

yes, actualy giving the docker-compose up/down command

You could verify the aliases in the container while it's running with: docker exec mail cat /etc/postfix/virtual.

Yes, it correctly contains:
alias1@domain  myuser

I am out of ideas whats going wrong in your setup. As @erik-wramner already wrote, you could enable debugging and check the relevant logs.
One last thing: You wrote myuser? Is that correct? In the examples I provided you have to use a complete email address such as<EMAIL_ADDRESS>Did you tried that: setup.sh alias add<EMAIL_ADDRESS><EMAIL_ADDRESS>If nothing else works, I would start again from scratch and strictly following the readme guide.

Now with
setup.sh alias add<EMAIL_ADDRESS><EMAIL_ADDRESS>
works, thank you. :+1:
I had to delete all previous aliases, like:
setup.sh alias del<EMAIL_ADDRESS>myuser


[BEAM-7923] Support docker screendiff integration tests

Added 2 more chrome options so that the interactive notebook screendiff integration tests work within docker.
Updated the golden screenshot.

Details:

By default, Docker runs a container with a /dev/shm shared memory space 64MB. This is typically too small for Chrome and will cause Chrome to crash when rendering large pages. To fix, run the container with docker run --shm-size=1gb to increase the size of /dev/shm. Since Chrome 65, this is no longer necessary. Instead, launch the browser with the --disable-dev-shm-usage flag
Without the chrome options, a test running within docker container will fail with following error:

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

R: @aaltay
PTAL, thx!

retest this please

retest this please

retest this please

retest this please

retest this please

The error seems to be affecting multiple precommits.
Created https://issues.apache.org/jira/browse/BEAM-9832 to track the issue.


unable to use w/ rem units

Describe the bug
I'm attempting to use rem units instead of px units to easily fit w/ my tailwind scales, and this seems to break my component as soon as I attempt to scroll within the scrollable frame.
Your minimal, reproducible example
https://codesandbox.io/p/sandbox/unruffled-hypatia-brmipn?file=%2Fsrc%2Fmain.tsx&selection=[{"endColumn"%3A13%2C"endLineNumber"%3A16%2C"startColumn"%3A13%2C"startLineNumber"%3A16}]
Steps to reproduce

Focus inside the scrollable div
Scroll

Expected behavior
N/A
How often does this bug happen?
None
Screenshots or Videos

Platform
macOS, Chromium 112.0.5615.49
tanstack-virtual version
3.0.0-beta.54
TypeScript version
5.0.4
Additional context
No response
Terms & Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I understand that if my bug cannot be reliable reproduced in a debuggable environment, it will probably not be fixed and this issue may even be closed.

Any luck on this? I'm having the same issue.

I don't think it is meant to work with any unit besides pixels. Better finer grain math under the hood. TranslateY is often works best as a standalone style separate from the rest of your CSS.


No response for FORWARDED PORTS

Version: 1.42.0 (user setup)
Commit: ae08d5460b5a45169385ff3fd44208f431992451
Date: 2020-02-06T10:51:34.058Z
Electron: 6.1.6
Chrome: 76.0.3809.146
Node.js: 12.4.0
V8: 7.6.303.31-electron.0
OS: Windows_NT x64 10.0.17134

CPUs
Intel(R) Core(TM) m3-6Y30 CPU @ 0.90GHz (4 x 1512)

GPU Status
2d_canvas: enabledflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmetal: disabled_offmultiple_raster_threads: enabled_onoop_rasterization: disabled_offprotected_video_decode: unavailable_offrasterization: enabledskia_renderer: disabled_offsurface_control: disabled_offsurface_synchronization: enabled_onvideo_decode: enabledviz_display_compositor: enabled_onviz_hit_test_surface_layer: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
3.91GB (0.45GB free)

Process Argv

Screen Reader
no

VM
0%

Remote
SSH: sxue

OS
Linux x64 4.9.168

CPUs
Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz (96 x 2699)

Memory (System)
376.78GB (3.37GB free)

VM
0%

Steps to Reproduce:

Connect to a remote server
Try to add a FORWARDED PORTS in Remote Explorer
No response

What port are you trying to forward?

@alexr00 there is even no input window when I click the add button.

Can you share a gif or screen shot? Are you using the "+" button?

Thanks! Can you open the dev tools and see if there's anything in the console?

@alexr00 Yep. When I click the "+"  button, there is an error in the console.

@axiqia thanks for helping track this down. Would you be able to test with the next(in 1-2 days) Insiders build to see if you still can't forward a port? Insiders installs side by side with stable, so you wouldn't lose anything by installing it. https://code.visualstudio.com/insiders/

@alexr00 I'd like to. Please notify me if the next build is released.

@axiqia We've had a couple build that contain the fix now. If you install Insiders you'll get a build with the fix. Thanks for testing this!

@alexr00 Thank you for fixing the bug. I have tested the fix. It works in my env.


crash on startup

https://gist.githubusercontent.com/natanfudge/453ba8e28d985a68f6ff69210fba9f49/raw/0b6706e9f8f875680739589c5a4745b09079753c/crash.txt

This is caused by a broken dependency in Conjuring, update it to 1.0.5 or newer


feat(query-core): log queryKey in createRetryer

@TkDodo As per our chat on Discord, I made a small change in the error that throws in createRetryer's onSuccess callback if the fetched data turns out to be undefined. The error now includes the affected queryKey - I have confirmed this in my personal project and successfully resolved the issue I had that way.
Open to any suggestions.
Thanks!

also you'd need to amend a test please:
FAIL packages/query-core/src/tests/query.test.tsx
  ● query › fetch should dispatch an error if the queryFn returns undefined

    expect(received).toMatchObject(expected)

    - Expected  - 1
    + Received  + 1

      Object {
    -   "error": [Error: Query data cannot be undefined],
    +   "error": [Error: Query data cannot be undefined - affected query key: ["query_30"]],
        "isError": true,
      }

      819 |     const error = new Error('Query data cannot be undefined')
      820 |
    > 821 |     expect(observerResult).toMatchObject({
          |                            ^
      822 |       isError: true,
      823 |       error,
      824 |     })

      at Object.<anonymous> (packages/query-core/src/tests/query.test.tsx:821:28)

I have applied your suggestions - all tests pass now locally.
Let me know if you need anything else.

Codecov Report
Base: 96.36% // Head: 96.57% // Increases project coverage by +0.21% :tada:

Coverage data is based on head (34e1322) compared to base (eab6e2c).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4213      +/-   ##
==========================================
+ Coverage   96.36%   96.57%   +0.21%
==========================================
  Files          45       72      +27
  Lines        2281     2949     +668
  Branches      640      814     +174
==========================================
+ Hits         2198     2848     +650
- Misses         80       99      +19
+ Partials        3        2       -1

Impacted Files
Coverage Δ

src/react/useQueries.ts

src/react/useIsMutating.ts

src/devtools/theme.tsx

src/react/QueryErrorResetBoundary.tsx

src/react/reactBatchedUpdates.ts

src/react/QueryClientProvider.tsx

src/core/hydration.ts

src/core/mutation.ts

src/react/logger.native.ts

src/core/query.ts

... and 107 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Updating bin stubs

Coverage remained the same at 55.997% when pulling 81c2e46d2c08d1a37d77834f8796578ccd8d3370 on update-bins into 40b4bd0e866cdba332677f2fa425432ea30b0781 on develop.


[CI][C++] TestDecimalFromReal for Float type failed in "AMD64 Windows MinGW MINGW32 C++"

Describe the bug, including details regarding any error messages, version, and platform.
The error messages is listed below:
[----------] 2 tests from TestDecimalFromReal/1, where TypeParam = std::pair<arrow::Decimal128, double>
[ RUN      ] TestDecimalFromReal/1.TestSuccess
D:/a/arrow/arrow/cpp/src/arrow/util/decimal_test.cc:766: Failure
Expected equality of these values:
  dec.ToString(scale)
    Which is: "4611686000000000000"
  expected
    Which is: "4611686018427387904"
D:/a/arrow/arrow/cpp/src/arrow/util/decimal_test.cc:766: Failure
Expected equality of these values:
  dec.ToString(scale)
    Which is: "-4611686000000000000"
  expected
    Which is: "-4611686018427387904"
D:/a/arrow/arrow/cpp/src/arrow/util/decimal_test.cc:766: Failure
Expected equality of these values:
  dec.ToString(scale)
    Which is: "9223372000000000000"
  expected
    Which is: "9223372036854775808"
D:/a/arrow/arrow/cpp/src/arrow/util/decimal_test.cc:766: Failure
Expected equality of these values:
  dec.ToString(scale)
    Which is: "-9223372000000000000"
  expected
    Which is: "-9223372036854775808"
D:/a/arrow/arrow/cpp/src/arrow/util/decimal_test.cc:766: Failure
Expected equality of these values:
  dec.ToString(scale)
    Which is: "18446744000000000000"
  expected
    Which is: "18446744073709551616"
D:/a/arrow/arrow/cpp/src/arrow/util/decimal_test.cc:766: Failure
Expected equality of these values:
  dec.ToString(scale)
    Which is: "-18446744000000000000"
  expected
    Which is: "-18446744073709551616"
[  FAILED  ] TestDecimalFromReal/1.TestSuccess, where TypeParam = std::pair<arrow::Decimal128, double> (0 ms)
[ RUN      ] TestDecimalFromReal/1.TestErrors
[       OK ] TestDecimalFromReal/1.TestErrors (0 ms)
[----------] 2 tests from TestDecimalFromReal/1 (0 ms total)

[----------] 2 tests from TestDecimalFromReal/2, where TypeParam = std::pair<arrow::Decimal256, float>
[ RUN      ] TestDecimalFromReal/2.TestSuccess
[       OK ] TestDecimalFromReal/2.TestSuccess (0 ms)
[ RUN      ] TestDecimalFromReal/2.TestErrors
[       OK ] TestDecimalFromReal/2.TestErrors (0 ms)
[----------] 2 tests from TestDecimalFromReal/2 (0 ms total)

[----------] 2 tests from TestDecimalFromReal/3, where TypeParam = std::pair<arrow::Decimal256, double>
[ RUN      ] TestDecimalFromReal/3.TestSuccess
D:/a/arrow/arrow/cpp/src/arrow/util/decimal_test.cc:766: Failure
Expected equality of these values:
  dec.ToString(scale)
    Which is: "4611686000000000000"
  expected
    Which is: "4611686018427387904"
D:/a/arrow/arrow/cpp/src/arrow/util/decimal_test.cc:766: Failure
Expected equality of these values:
  dec.ToString(scale)
    Which is: "-4611686000000000000"
  expected
    Which is: "-4611686018427387904"
D:/a/arrow/arrow/cpp/src/arrow/util/decimal_test.cc:766: Failure
Expected equality of these values:
  dec.ToString(scale)
    Which is: "9223372000000000000"
  expected
    Which is: "9223372036854775808"
D:/a/arrow/arrow/cpp/src/arrow/util/decimal_test.cc:766: Failure
Expected equality of these values:
  dec.ToString(scale)
    Which is: "-9223372000000000000"
  expected
    Which is: "-9223372036854775808"
D:/a/arrow/arrow/cpp/src/arrow/util/decimal_test.cc:766: Failure
Expected equality of these values:
  dec.ToString(scale)
    Which is: "18446744000000000000"
  expected
    Which is: "18446744073709551616"
D:/a/arrow/arrow/cpp/src/arrow/util/decimal_test.cc:766: Failure
Expected equality of these values:
  dec.ToString(scale)
    Which is: "-18446744000000000000"
  expected
    Which is: "-18446744073709551616"
[  FAILED  ] TestDecimalFromReal/3.TestSuccess, where TypeParam = std::pair<arrow::Decimal256, double> (0 ms)
[ RUN      ] TestDecimalFromReal/3.TestErrors

(1) (3) in TestDecimalFromReal are all float. So I guess there would be something wrong when handling float type. The CI can be seen here: https://github.com/apache/arrow/actions/runs/4988181428/jobs/8930665332?pr=35605
Component(s)
C++, Continuous Integration

@pitrou Mind take a look?

These tests haven't changed in a while AFAIK, and they used to pass on MinGW. If possible it would be nice to research the GHA history and find out at which point they started failing.

Yes, https://github.com/apache/arrow/issues/35571 is also MinGW-related. I guess they have similiar reason. I submit a hotfix but I guess maybe we'd better find the root cause

Where is your hotfix?

( https://github.com/apache/arrow/pull/35605 . Just for float point and parquet decryption, not for this issue )

I tried this on my MSYS2 environment but this is not reproduced...
Anyway, I'll take a look at this.

Is this closed by #35605 / #35571 ?

Is this closed by https://github.com/apache/arrow/pull/35605 / https://github.com/apache/arrow/issues/35571 ?

@westonpace I think they're different problems

// Common tests for Decimal128::FromReal(T, ...) and Decimal256::FromReal(T, ...)
template <typename T>
class TestDecimalFromReal : public ::testing::Test {
 public:
  using Decimal = typename T::first_type;
  using Real = typename T::second_type;
  using ParamType = FromRealTestParam<Real>;

  void TestSuccess() {
    const std::vector<ParamType> params{
        // clang-format off
        {0.0f, 1, 0, "0"},
        {-0.0f, 1, 0, "0"},
        {0.0f, 19, 4, "0.0000"},
        {-0.0f, 19, 4, "0.0000"},
        {123.0f, 7, 4, "123.0000"},
        {-123.0f, 7, 4, "-123.0000"},
        {456.78f, 7, 4, "456.7800"},
        {-456.78f, 7, 4, "-456.7800"},
        {456.784f, 5, 2, "456.78"},
        {-456.784f, 5, 2, "-456.78"},
        {456.786f, 5, 2, "456.79"},
        {-456.786f, 5, 2, "-456.79"},
        {999.99f, 5, 2, "999.99"},
        {-999.99f, 5, 2, "-999.99"},
        {123.0f, 19, 0, "123"},
        {-123.0f, 19, 0, "-123"},
        {123.4f, 19, 0, "123"},
        {-123.4f, 19, 0, "-123"},
        {123.6f, 19, 0, "124"},
        {-123.6f, 19, 0, "-124"},
        // 2**62
        {4.611686e+18f, 19, 0, "4611686018427387904"},
        {-4.611686e+18f, 19, 0, "-4611686018427387904"},
        // 2**63
        {9.223372e+18f, 19, 0, "9223372036854775808"},
        {-9.223372e+18f, 19, 0, "-9223372036854775808"},
        // 2**64
        {1.8446744e+19f, 20, 0, "18446744073709551616"},
        {-1.8446744e+19f, 20, 0, "-18446744073709551616"}
        // clang-format on
    };
    for (const ParamType& param : params) {
      CheckDecimalFromReal<Decimal>(param.real, param.precision, param.scale,
                                    param.expected);
    }
  }

Here meets:
        // 2**62
        {4.611686e+18f, 19, 0, "4611686018427387904"},
        {-4.611686e+18f, 19, 0, "-4611686018427387904"},
        // 2**63
        {9.223372e+18f, 19, 0, "9223372036854775808"},
        {-9.223372e+18f, 19, 0, "-9223372036854775808"},
        // 2**64
        {1.8446744e+19f, 20, 0, "18446744073709551616"},
        {-1.8446744e+19f, 20, 0, "-18446744073709551616"}

These values would be truncated on MinGW when type is float

I could reproduce this.


expose only the positions of those brackets colored by native bracket pair colorization

Question: I know exposing tokens to extensions is a headache, but is it now possible and non-headache to expose only the positions of those brackets colored by native bracket pair colorization?
like this:
{type: "{" | "}" | "(" | ")" | "[" | "]"; line: number; column: number; }[]
or like this:
{type: "{" | "}" | "(" | ")" | "[" | "]"; globalStringIndex: number; }[]
No tree structure needed, just a simple array of positions. It will be super huge speed upgrade for Blockman (already 19k installs).
Originally posted by @leodevbro in https://github.com/microsoft/vscode/issues/128465#issuecomment-900604355

Oh, thanks so much for posting this feature request.

Syncing over all bracket positions might be headache.
However, I can imagine an async API that you can ask to query bracket pairs in a given range.
It could be limited to return a certain amount of bracket pairs (e.g. 1k).
I think bracket pair positions can also be encoded very efficient in some UInt32Array, so there is not much overhead (5 values per pair, so at most 20kb).
We don't know the globalStringIndex (only compute line/column for faster queries).
That's roughly what I have in mind:
/**
 * Returns undefined if bracket pair information are not available (e.g. because the feature is disabled or the document is too large).
 */
async provideBracketPairs(model: TextModel, range: Range): Promise<BracketPairs | undefined>;

interface BracketPairs {
    pairs: BracketPair[];
    truncated: boolean; // Indicates whether there are more positions that are not included for performance reasons
    /**
     * Indicates if bracket pairs changed in the meantime.
     */
    inaccurate: boolean;
}

interface BracketPair {
    openRange: Range;
    closeRange: Range;
}

Would that be helpful? I don't know all the implications of this async API for your usecase.

If that approach is significantly faster than the Bracket Pair Colorizer 2 approach (Which is also async, and it's implemented in Blockman), then I guess it's worth making.
Just one note:
bracket number limit e.g. 1k is maybe fine, well, probably we should maximize this limit as big as possible, like 10k maybe.
Well, I guess most of real world code files does not have more than 3k brackets even if they have 10 thousand lines.
But probably 10k (or more) bracket limit would be good just in case.
One more note:
Whatever the limit, 1k, 5k or 10k, I think Blockman cannot work with queried brackets of given range, it needs to know the entire file brackets, because the rendered blocks of current viewport window often can depend on the brackets which are located at the last line of the file or at the first line of the file, or both.
Well, for this one, I guess there is simple solution. If we have an extreme edge case like a file with 20k brackets, then the bracket query should return nothing. It will not be a deal breaker, because most of developers don't work with such huge files.

Whatever the limit, 1k, 5k or 10k, I think Blockman cannot work with queried brackets of given range, it needs to know the entire file brackets, because the rendered blocks of current viewport window often can depend on the brackets which are located at the last line of the file or at the first line of the file, or both.

Hmm, do you really need all bracket pairs?
What about all those pairs whose range intersects a given range?
Consider this:
{}
{
    {
        // start: a lot of bracket pairs here
        {}{}{}{}{}{}{}{}{}{}{}{}{}
        {}{}{}{}{}{}{}{}{}{}{}{}{}
        // end
    }

    // ...

    {
    // start of viewport
    }
    {
        {}
    }
    {
    // end of viewport
    }
}

Do you really need the brackets in "a lot of bracket pairs here"?

It's pretty cool approach for optimization.
Well, yeah, if it is doable with not a huge pain, then
querying only the pairs whose range intersects a given viewport range
is very cool.

Got a question about the statement:

Extensions expect the tokens to be correct. However, with the current state of textmate and due to some performance optimizations, we cannot guarantee correctness.

It's from https://github.com/microsoft/vscode/issues/128465
I see there are some feature updates about native bracket pair colorization. So, now, does the new version of VSCode guarantee correctness of bracket tokens? If no, then does querying only the pairs whose range intersects a given viewport range solve this problem? I mean, if the range will be limited with the viewport range, then the number of bracket tokens will always be small, so it means there will be no need for any more performance optimization, and it means there will be no more sacrifice of correctness.
@hediet,
Also, I would like to ask you, in what state is the feature idea of querying only the pairs whose range intersects a given viewport range? Is the VSCode team planning to implement it? Maybe in two months? Maybe in three months?

The problem is this:
const content = this.document.getText();
const bracketPairs = await vscode.languages.getBracketPairs(this.document.uri, this.getViewPortRange());
// use content and bracket pairs to create decorations

How can we prevent someone from doing this?
The issue here is that getBracketPairs is async. When the call returns, the text buffer could have already been modified and not agree with content anymore.

getBracketPairs is async. When the call returns, the text buffer could have already been modified and not agree with content anymore.

In case of Blockman extension, I guess currently it works basically that way, when typing text on keyboard Blockman cannot analyze updated content quickly enough to keep up the changes in real time, so it deals with timers. On each text change event, Blockman starts re-analyzing the file after t seconds. If we set the t to be 0 seconds, the editor UI will most likely freeze during typing some text on the keyboard, so, that's why the default time is 1.2 seconds. The freezing is still noticeable during typing, but it's not a deal breaker for many coders because it feels rare and short.
Note: the timer is triggered only for the last change event and is not triggered on every event during the 1.2 seconds, so it will not get overloaded and bounced.
So, yeah, with Blockman, if text changes during the ongoing analyzing/rendering process, the blocks will be rendered for the old text, so it will look incorrect visually, but Blockman detects the change events and re-renders the blocks again with the new content. So the blocks finally are being rendered correctly. So, I guess the the blocks appear incorrect for about 1 second (average time) when working on a 5,000 line file, but if it will have access to the native tokens of brackets, I think this average time will go down to maybe 0.1 seconds, and so it will feel more comfy.

Are there any related investments?


Create main.py

This will let you execute python -m binwalk directly
This is mainly for Windows Users since executing binwalk directly doesn't work for some people. It is an easy fix with nothing added, just replicated.

I effectively did the same thing, but updated scripts/binwalk so there wasn't code duplication. 👍


redshift optimization

Description

Redshift Table lock issue while pipeline runs solved
Redshift destination Performance increased

[ ] Test A
[ ] Test B

Checklist

[ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation)
[ ] I have performed a self-review of my own code
[ ] I have added unit tests that prove my fix is effective or that my feature works
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] If new documentation has been added, relative paths have been added to the appropriate section of docs/mint.json

cc:

can you describe what tests you ran to verify this change works?

I have done testing below
1.data integration pipeline with redshift as destination
2. Tried with duplicate keys
3. At the time of pipeline runs trying select querying with same table

@dy46 is there any suggestions here ?

@tommydangerous @dy46


Added constructors to allow multiple running instances of FFmpeg

When doing batch conversions on multi-core computers, it takes less time to have multiple instances of FFmpeg running concurrently, which prior to this PR, was explicitly disabled.  Now it's only disabled by default.

@evandixon Have you had success with this fork of the library? We are also using a fork of MediaToolkit and running into some concurrency issues currently in tests (since we have multiple tests that use this library running in parallel with xunit). The error we seem to be getting when there is concurrent access is System.ComponentModel.Win32Exception : Access is denied.
I am thinking of pulling your changes into our fork as well, but wanted to check with you first to make sure it is working as expected for you, and to check and see if you have made any additional changes that are not in this pull request.
Thanks,
Dan Ludwig

@danludwig Since I originally made this PR, I've started doing similar things for other tools.  I've just pushed another commit that should fix additional concurrency issues.
My original method of concurrent execution unfortunately overlooked exceptions caused by concurrent execution.  I've since fixed it so that I'm notified when any exceptions are thrown.  My latest commit has been tested in two ways:

Concurrently executing Engine.Convert with a single instance of Engine
Concurrently executing Engine.Convert with each execution using a unique instance of Engine


RequestDeviceAsync always return null on Android 13

Hi, I've tried to use your library to test a Bluetooth app. I'm currently trying your Battery level sample on Android device (API 33, BluetoothLE nuget last version).
After Adding activity correctly and right permissions, it still not work.
var device = await Bluetooth.RequestDeviceAsync(new RequestDeviceOptions { AcceptAllDevices = true });
This method always return null. With a near device or connected device it's the same. It return null on android. But it work on Windows 11 with the same devices.
I've tried the ScanForDevicesAsync method but even if i add Scan permission in manifest i got an user permission exception...
here is permissions :

What's wrong ?
Thank you for help !

It's a permission issue, We don't prompt for the BLUETOOTH_CONNECT permission (proximity devices) and the RequestDeviceAsync() method doesn't prompt like said in method summary... Maybe it doesn't work on android 13
For the SCAN permission, by enabling it in app permission on Device, it work now !

I've added #344 to track the issue where the library doesn't prompt for permissions when required.

I've made it easier to prompt for permission in Xamarin/.NET MAUI projects with this add-on library:-
https://inthehand.com/2023/10/11/asking-for-permission/

I've try your solution few days ago now and it work on all my platforms!
Thank you very much for your support and your work!!! 🙏


Migrate sense uploader to use backend listener

Did you test it well? And especially from non-gui environment, like Jenkins?

Yes, I tested it on Jenkins, and then compare results at Sense and Jenkins Performance Plugin. They're equals.


Jellyfin_HA

Hello Koying
I just downloaded Jellyfin via HASC for HA"Home Assisanta"
I have 0 clue how to there no info when download Iv looked into ever folder and couldn't work it out
I would love for this to be more of a thing but sadly i don't think it will become much iu have got some what another type working but there not status or play / pause etc
also only thing missing on the card I'm using is swipe up to change etc
but anyways would love a how to you can also check out how i did What I'm so far using and try it out or become somewhat the same
https://emby.media/community/index.php?/topic/79239-what-is-the-home-assistant-add-on/
I though emby would of been the same as jellyfin even know it worked parts of it didn't

You just install that repository in HACS in HA, then restart HA and you'll get then a new integration avalaible in HA named Jellyfin that you can setup straight in HA (it just needs IP of Jellyfin server and some valid credentials on it !

Hey, have you been able to fix this issue ? I just downloaded this integration through HACS, restart twice, and still no Jellyfin in the integrations. Maybe something broke since the last release in July ?


use correct framework

the undocumented names (without period in dotnet 5+) can drop their support when move to double-digit (to avoid potential conflicts) learn.microsoft.com/en-us/dotnet/standard/frameworks#supported-target-frameworks

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for the pr!


GPU race conditions from tf.map_fn

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): All the code that caused this issue uses Tensorflow/Keras operations.
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 1.14.0
Python version: 3.6.8
CUDA/cuDNN version: 10.0/7.6.2
GPU model and memory: RTX 6000x2, 48 GB

Describe the current behavior
I've created a custom layer called ROI in Keras that uses tf.map_fn, precisely because it has unknown parameter that it needs to take as tensor object.
This layer works perfectly on CPU inference and training, it also works perfectly on GPU during inference. But during training with a powerful GPU, exception about GPU colocation of ROI layer occurs:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation ROI/map/while/Identity_1: Could not satisfy explicit device specification '' because the node node ROI/map/while/Identity_1 (defined at /path/to/custom/layer/custom.py:70) placed on device No device assignments were active during op 'ROI/map/while/Identity_1' creation.

[[node ROI/map/while/Identity_1 (defined at /path/to/custom/layer/custom.py:70) Additional information about colocations: No node-device colocations were active during                                                                                op 'ROI/map/while/Identity_1' creation.
No device assignments were active during op 'ROI/map/while/Identity_1' creation.`

Manual colocation of ROI layer to CPU device with tf.device worked, but I want ROI to support GPU as well.
My hypothesis
ROI layer works on CPU because only single core at a time should handle the layer - even if multiprocessing is activated, there are few cores slowly balancing the task.
But whenever GPU is utilized, thousands of cores are working together in parallel and they are not waiting for each other to finish their tasks. Thus one of the processes tries to gather data from TensorArray that is still in while loop (using tf.map_fn), which causes the error.
Describe the expected behavior
Tensorflow should be able to handle these race conditions by waiting for its own tf.map_fn to be finished instead of raising exceptions.
Code to reproduce the issue
This is the code
 that instantly causes the mentioned issue on my local machine.
Other info / logs
Full Log:
Traceback (most recent call last):
  File "/path/to/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/path/to/site-packages/tensorflow/python/client/session.py", line 1339, in _run_fn
    self._extend_graph()
  File "/path/to/site-packages/tensorflow/python/client/session.py", line 1374, in _extend_graph
    tf_session.ExtendSession(self._session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation ROI/map/while/Identity_1: Could not satisfy explicit device specification '' because the node {{colocation_node ROI/map/while/Identity_1}} was colocated with a group of nodes that required incompatible device '/job:localhost/replica:0/task:0/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0, /job:localhost/replica:0/task:0/device:GPU:0].

We are working on reproducing it through a Google Colab notebook to share with you. Not sure if the issue appears only with powerful multi GPUs or also with average GPUs like the default K80 from colab. We will share updates ASAP @oanush, @jvishnuvardhan

Update
Tried to implement the slightly altered code to Google Colab and couldn't replicate the issue. Then tried the same slightly altered code on my machine and surprisingly the issue still couldn't be replicated, before I pre-loaded the weights before training - then issue immediately occurred back up (on my local machine, and not colab).
Thus there are two prerequisites to replicate this issue:

Load arbitrary of .h5 weights before compiling the network with ROI layer implemented within.
Execute the training script on a powerful GPU. K80 from Colab fails to replicate this issue for some reason, but I've tested it on RTX 6000x2, P5000, P4000 and GTX 960 and they successfully replicate the issue if condition in the first bullet point is satisfied (although GTX 960 might fail sometimes when it's overloaded).

By the way, I highlighted word arbitrary in the first bullet point because this happens for any weight. Thus you can simply train the network, save its weights and then try training again by pre-loading weights with model.load_weights before model.compile.

Currently I'm minimizing the code as much as possible, and I believe I'll be able to send it tomorrow. I think it'll make it easier to replicate this issue (although it is important to consider the usage of powerful GPU). I'll be probably able to post minimized code by tomorrow.
Otherwise, I'm confident that satisfying mentioned prerequisites will replicate this issue.
Thank you for the response! @oanush @jvishnuvardhan

Could not reproduce the issue with normal CPU and GPU run-time, also not able to reproduce issue for the GPU model mentioned.

I've been able to construct a minimal code that causes this issue instantly on my local machine, you can see it here (I've also updated the issue with this link). Please make sure to execute this script twice, i.e python GPU_ROI_issue.py;python GPU_ROI_issue.py so that it saves the weights after training and then tries to train with loaded weights which causes the error mentioned in this issue.
Also as mentioned earlier, please make sure that this code is executed on efficient GPU, since from my experience it is only caused on them.
Thank you!

Update
No fix is yet found for training a model containing layer that uses tf.map_fn on GPU with pre-loaded weights. As soon as any weights of any layers are manually set (i.e with layer.set_weights or model.load_weights), which uses batch_set_value to pass them into the feed_dict to a symbolic callable function that Keras creates (but kernel/bias initializers of layers work perfectly though).
Perhaps channels_first (NCHW) setting of layers might have something to do with it, considering that Tensorflow has a default channel order of NHWC and therefore is probably optimized for that exact format.
The only workaround with tf.map_fn is, by colocating ROIPoolingLayer on CPU device with tf.device("CPU:0"), but this is obviously not an optimal solution (considering that GPU support is highly significant for efficiency).
Another obvious workaround is using vectorized operations only, but this might not be an easy task for such layer without hardcoding.

@oanush @jvishnuvardhan were you able to reproduce the issue?

Could not reproduce the issue with normal CPU and GPU run-time, also not able to reproduce issue for the GPU model mentioned.

I've been able to construct a minimal code that causes this issue instantly on my local machine, you can see it here (I've also updated the issue with this link). Please make sure to execute this script twice, i.e python GPU_ROI_issue.py;python GPU_ROI_issue.py so that it saves the weights after training and then tries to train with loaded weights which causes the error mentioned in this issue.
Also as mentioned earlier, please make sure that this code is executed on efficient GPU, since from my experience it is only caused on them.
Thank you!

Update
No fix is yet found for training a model containing layer that uses tf.map_fn on GPU with pre-loaded weights. As soon as any weights of any layers are manually set (i.e with layer.set_weights or model.load_weights), which uses batch_set_value to pass them into the feed_dict to a symbolic callable function that Keras creates (but kernel/bias initializers of layers work perfectly though).
Perhaps channels_first (NCHW) setting of layers might have something to do with it, considering that Tensorflow has a default channel order of NHWC and therefore is probably optimized for that exact format.
The only workaround with tf.map_fn is, by colocating ROIPoolingLayer on CPU device with tf.device("CPU:0"), but this is obviously not an optimal solution (considering that GPU support is highly significant for efficiency).
Another obvious workaround is using vectorized operations only, but this might not be an easy task for such layer without hardcoding.

@oanush @jvishnuvardhan Channel order is not a problem, I've set all layers to channels_last (NHWC) order and modified ROI pooling layer to support this change. I've also replaced every keras usage with tf.keras (to ensure this is not a compatibility issue) - and I can still reproduce this error.
This is the code using tf.keras and NHWC layer order that reproduces the issue mentioned in the original post

I can reproduce it with a 1080 Ti.
The problem is not a race condition. Part of the constructed graph is not GPU-friendly and it has to be placed on the CPU. The placer manages to work out a good arrangement if the model has not been loaded any weights. Otherwise, something somewhere takes the wrong turn and the placer arrives to a pathological state.
I tried to compare graph dumps with and without weight load, but they are exceedingly complicated and the reason for the mismatch is not apparent. Although it looks like the run with weight load is missing all the backprop-related nodes until a late point. For example, without weight load, placer_input_2.pbtxt is 325 kb and it has lots of "training/..." nodes. With weight load, placer_input_2.pbtxt and it has no "training" nodes at all. (They only appear in placer_input_4.pbtxt.) In placer_input_2.pbtxt, the node roi_layer/map/while/TensorArrayWrite/TensorArrayWriteV3 gets assigned to the CPU without weight load and to the GPU with weight load.

I can reproduce it with a 1080 Ti.
The problem is not a race condition. Part of the constructed graph is not GPU-friendly and it has to be placed on the CPU. The placer manages to work out a good arrangement if the model has not been loaded any weights. Otherwise, something somewhere takes the wrong turn and the placer arrives to a pathological state.
I tried to compare graph dumps with and without weight load, but they are exceedingly complicated and the reason for the mismatch is not apparent. Although it looks like the run with weight load is missing all the backprop-related nodes until a late point. For example, without weight load, placer_input_2.pbtxt is 325 kb and it has lots of "training/..." nodes. With weight load, placer_input_2.pbtxt is 116 kb and it has no "training" nodes at all. (They only appear in placer_input_4.pbtxt.) In placer_output_2.pbtxt, the node roi_layer/map/while/TensorArrayWrite/TensorArrayWriteV3 gets assigned to the CPU without weight load and to the GPU with weight load.

Hello, thank you for the attempt of reproduction and debugging info. I think race conditions or some GPU related issues might be precisely caused due to part of constructed graph not being GPU-friendly as you mentioned (I assume this implies tf.map_fn compatibility, or at least some part of its code), but I'm not aware of exact issue.
Manual colocation (tf.device wrapper) completely fixes this issue by colocating ROI graph to CPU - although I don't think this is as efficient as Tensorflow's automatic colocator. But I still find weight loading concept quite perplexing (and its relations with device colocations), thank you very much for the experiment though, it still has very interesting results.

More information:

During placement, in core/common_runtime/colocation_graph.cc, placer creates node 93

{name:'roi_layer/map/while/map/TensorArray_1' id:93 op device:{/job:localhost/replica:0/task:0/device:GPU:0} def:{{{node roi_layer/map/while/map/TensorArray_1}} = TensorArrayV3[clear_after_read=
true, dtype=DT_FLOAT, dynamic_size=false, element_shape=<unknown>, identical_element_shapes=true, tensor_array_name=""](roi_layer/map/while/map/strided_slice)}}

and assigns it to a Member
(Member(assigned_device
_name_index_=-1 requested_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' assigned_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' resource_device_name_='' supported_device_types_=[GPU,  CPU, XLA_CPU, XLA_GPU] possible_devices_=[])

And it assigns node 588
{name:'training/SGD/gradients/roi_layer/map/while/map/TensorArrayStack/TensorArrayGatherV3_grad/TensorArrayGrad/TensorArrayGradV3/StackPushV2' id:588 op device:{} def:{{{node training/SGD/gradie
nts/roi_layer/map/while/map/TensorArrayStack/TensorArrayGatherV3_grad/TensorArrayGrad/TensorArrayGradV3/StackPushV2}} = StackPushV2[T=DT_RESOURCE, _class=["loc:@roi_layer/map/while/map/TensorArray_1"], swap_me
mory=false](training/SGD/gradients/roi_layer/map/while/map/TensorArrayStack/TensorArrayGatherV3_grad/TensorArrayGrad/TensorArrayGradV3/Enter, roi_layer/map/while/map/TensorArray_1, ^training/SGD/gradients/Add)
}}

to member
(Member(assigned_device_name_index_=-1 requested_device_name_='' assigned_device_name_='' resource_device_name_='' supported_device_types_=[CPU] possible_devices_=[])```

- so far so good.

Then the placer does a lot of optimizations and eventually gets around to colocate node 93 and node 588:

Colocating 93 {name:'roi_layer/map/while/map/TensorArray_1' id:93 op device:{/job:localhost/replica:0/task:0/device:GPU:0} def:{{{node roi_layer/map/while/map/TensorArray_1}} = TensorArrayV3clear_after_read=true, dtype=DT_FLOAT, dynamic_size=false, element_shape=, identical_element_shapes=true, tensor_array_name=""}} and 588 {name:'training/SGD/gradients/roi_layer/map/while/map/TensorArrayStack/TensorArrayGatherV3_grad/TensorArrayGrad/TensorArrayGradV3/StackPushV2' id:588 op device:{} def:{{{node training/SGD/gradients/roi_layer/map/while/map/TensorArrayStack/TensorArray
GatherV3_grad/TensorArrayGrad/TensorArrayGradV3/StackPushV2}} = StackPushV2[T=DT_RESOURCE, class=["loc:@roi_layer/map/while/map/TensorArray_1"], swap_memory=false](training/SGD/gradients/roi_layer/map/while/map/TensorArrayStack/TensorArrayGatherV3_grad/TensorArrayGrad/TensorArrayGradV3/Enter, roi_layer/map/while/map/TensorArray_1, ^training/SGD/gradients/Add)}}
New root: 0x55a18977e448 Member(assigned_device_name_index=-1 requested_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' assigned_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' resourc
e_device_name_='' supported_device_types_=[GPU, CPU, XLA_CPU, XLA_GPU] possible_devices_=[]
MergeDeviceNames(0x55a18977e448 0x55a1897bcdb0): requested /job:localhost/replica:0/task:0/device:GPU:0
0x55a18977e448: merging devices with 0x55a1897bcdb0
New root: 0x55a18977e448 Member(assigned_device_name_index_=-1 requested_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' assigned_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' resource_device_name_='' supported_device_types_=[CPU] possible_devices_=[]

And now we end up with a Member object that was _requested_ to be on the GPU, but can only be _placed_ on the CPU. (Because colocation logic does not check for this possibility.)

Then, quite a while later, the problem is noticed and reported in a singularly unhelpful error message.

To me this looks like a bug in the placer.

Okay, I think I see the big picture.
We have a graph containing an operation that can be placed on the GPU, but whose gradient can only be placed on the CPU.
When constructed normally via model.compile/model.fit, it works because the graph is completely constructed in one go.
When we load weights, the graph is partially constructed inside K.batch_set_value, using only a subset of the nodes. Since gradient ops are still missing, the placer has no problem placing everything on the GPU.
When we then try to train, we add the remaining ops, but we retain the device designations made during the partial construction, which are incompatible with the full construction.
Or something like that. :)

Okay, I think I see the big picture.
We have a graph containing an operation that can be placed on the GPU, but whose gradient can only be placed on the CPU.
When constructed normally via model.compile/model.fit, it works because the graph is completely constructed in one go.
When we load weights, the graph is partially constructed inside K.batch_set_value, using only a subset of the nodes. Since gradient ops are still missing, the placer has no problem placing everything on the GPU.
When we then try to train, we add the remaining ops, but we retain the device designations made during the partial construction, which are incompatible with the full construction.
Or something like that. :)
P.S. And I found where exactly device designations are being retained. Right here:
https://github.com/tensorflow/tensorflow/blob/ee16fcac960ae660e0e4496658a366e2f745e1f0/tensorflow/core/common_runtime/graph_execution_state.cc#L243
After commenting out that line as well as these
https://github.com/tensorflow/tensorflow/blob/ee16fcac960ae660e0e4496658a366e2f745e1f0/tensorflow/core/common_runtime/direct_session.cc#L1502-L1505
, and then rebuilding, the error goes away!
( Of course, those lines are probably there for a reason and simply axing them in the live repository is probably not the optimal solution ... )

That's very interesting, so subset of regular operations that are temporarily held with K.batch_set_value during weight assignment have complete support for GPU - therefore it is assumed that the whole graph, including the complement set containing gradient operations should be colocated on GPU device (however, not all gradient operations in this complement set have GPU kernels).
I still don't get why automatic weight initializers and load_weights use similar graph construction and colocation schemes though, wouldn't it be easier? (perhaps not, but not sure why).
Regardless, thank you for your findings. I hope there is at least slightly simple general solution for this weird issue.

therefore it is assumed that the whole graph, including the complement set containing gradient operations should be colocated on GPU device

I don't know if it's assumed, or simply that no one yet realized this scenario is possible.

I still don't get why automatic weight initializers and load_weights use similar graph construction and colocation schemes though, wouldn't it be easier? (perhaps not, but not sure why).

As far as I understand, when the graph is initially constructed, weight tensors are provided with initializers. They are told "if you are asked to execute but you don't have any contents, go to this RNG and pull some values out of it to fill yourself." But nothing actually happens until you start the session run inside model.fit().
When you load weights, you explicitly assign contents to the tensors, which requires a separate session run.
What I'm not sure about is why you don't get the complete graph right out of model.compile(), since that would solve the problem too.
FYI, on a tangential topic, in case you haven't realized. On your system, there are (probably) two separate keras Python source trees, one in "keras", and one in "tensorflow.keras". They aren't necessarily the same (they are very similar but may have different versions.)  If you pull the code from the tensorflow repository right now and follow the instructions to build it from source, you'll get the new binaries, but your code will still be using the same python code.

I still don't get why automatic weight initializers and load_weights use similar graph construction and colocation schemes though, wouldn't it be easier? (perhaps not, but not sure why).

As far as I understand, when the graph is initially constructed, weight tensors are provided with initializers. They are told "if you are asked to execute but you don't have any contents, go to this RNG and pull some values out of it to fill yourself." But nothing actually happens until you start the session run inside model.fit().
When you load weights, you explicitly assign contents to the tensors, which requires a separate session run.
What I'm not sure about is why you don't get the complete graph right out of model.compile(), since that would solve the problem too.

Yes, what I mentioned was that it could be more explicit if random initializers created separate session to fill the weight tensors just like manual methods. But I agree, whole graph construction during model.compile would fix this issue as well - I guess there's some drawback to that though.

FYI, on a tangential topic, in case you haven't realized. On your system, there are (probably) two separate keras Python source trees, one in "keras", and one in "tensorflow.keras". They aren't necessarily the same (they are very similar but may have different versions.) If you pull the code from the tensorflow repository right now and follow the instructions to build it from source, you'll get the new binaries, but your code will still be using the same python code.
To use the "correct" keras, you need to add "tensorflow." to all import directives.
import tensorflow.keras as keras
from tensorflow.keras import callbacks
import tensorflow.keras.layers as layers
from tensorflow.keras.activations import linear

etc

Yes, I am aware of that. In fact I've linked a new code in one of my replies for this issue, that uses tf.keras and NHWC format (I was hoping it would fix this problem but it didn't)

FYI2, for a hacky workaround for this problem that does not involve manually holding the ROI graph to the CPU, you can run the training for 1 epoch, then load weights, then train for however long you need.

I don't think I understand how can these steps be executed. Can I train the model for 1 epoch and then load weights without stopping this process? I guess I need to temporarily change things in source code?

                          epochs=1,
                          use_multiprocessing=False,
                          workers=1)
     if os.path.exists(weights_file):
        model.load_weights("new.h5")
     history = model.fit_generator(TrainSequence(data_set),
                          epochs=10,
                          use_multiprocessing=False,
                          workers=1)

Pretty sure it would work

@reedwm could you help to take a look at this?

@ekuznetsov139: I got a similar problem and I confirmed your trick of fit - load_weights and then fit again works. Thank you very much!

Hi There,
We are checking to see if you still need help on this, as you are using an older version of tensorflow which is officially considered end of life . We recommend that you upgrade to the latest 2.x version and let us know if the issue still persists in newer versions. Please open a new issue for any help you need against 2.x, and we will get you the right help.
This issue will be closed automatically 7 days from now. If you still need help with this issue, please provide us with more information.


Brainstorm what user-centered form could be w/ end of Jan deadline

Patrick and Birkin met to talk about weekly developments but still need to meet talk about long term minimal end of January functionality

New agenda link
[https://docs.google.com/document/d/10HHr9JX_yiuvloqRYX2ehUsAaOP4dEwO62NZet4ICM0/edit]

Officially closed


The array  'columns' is not properly closed on the example.

The array  'columns' is not properly closed on the example.

Thanks, merged.


[Feature request] Add support for bibtex compilation

It would be great to have an option to compile bibtex references together with the whole document. In essence, a convention that the .bib file should be called the same as the .tex file would be sufficient I think.
As for now I post-compile .aux file with bibtex manually to make the whole thing working.
The general pipeline would be the following (source):
pdflatex <options> doc.tex &&
bibtex doc && # compiles doc.bib
pdflatex && <options> doc.tex
pdflatex <options> doc.tex

Though I think adding it as a permanent pipeline might break the compilation process in case there is no .bib file so it definitely should be an option.
I would try to tuckle the problem myself but I doubt I will succeed in a short term as I'm not a vim-script ninja.
P.S. Sadly, there is no feature-request label to add

I created the branch bibtex and made a commit: ca7ab59bdbecc012b09e96ac9dd7cef7f2085041
If you set the option let g:neotex_bibtex=1, it will check if there is a .bib file in the same folder and with the same name as the .tex file, when loading a latex file. If there is a .bib file it will add the additional bibtex and pdflatex commands to the pipeline.
This way you can keep g:neotex_bibtex enabled on tex files without bibtex.
It should also work using latexdiff.
So it passed the "It works for me"-test, but maybe you can test yourself and give me feedback.

Great, thanks! I'll try to test it in the following days and get back to you.

My apologies for the fact testing took a little bit more then a month (basically it turns out that only now I had a necessity and time to get back to bibtex related stuff).
Sadly compilation with bibtex still do not work for me. I've pulled from bibtex branch and added let g:neotex_bibtex = 1 to my neovim config though nothing really changed.
How can I assist you to simplify the debugging process? Any kind of logs I can provide to you?


updated interested-technical-writers.md

Added my profile for interest in technical writers 2023

Thank you for your interest in Season of Docs. Please add your information to the bottom of the file as specified in the instructions, and verify that it does not break the table layout before you commit.


remove ! from links in README

Closes #2, I think. I'm not sure if these are supposed to have images but it looks like they're just a link, but with a ! to cause the broken image symbol to show up.
If there's some logo that's supposed to be there it should presumably look like the badge links at the top, i.e. something like:
[![name](link to NumFOCUS logo)NumFOCUS](https://numfocus.org)

Thanks!


Add docs for imagePullSecrets

@shawndwells is this what you were looking for?

Going to merge this. Thanks!


How to add data to specific storeName?

set, get seems a practice when using local storage and session storage. But i want to use WebSql driver. I became limited to add and found some difficulties on getting data through that way/system. Can you give me an example on how to properly add data in a "storeName"( or table like) using LocalForage?

Hi there.
You should either use the config() method to set the storeName for your default LF instance, or use createInstance() (which accepts all the options as config()) and create a new LF instance to manage the specific storeName.
Closing this for now. Feel free to re-open in any case.

Im using Typescript, following steps here https://ionicframework.com/docs/storage/
when i use this I am limited to utilize method of LF for not declare in typescript. I can only setup once here
IonicModule.forRoot(MyApp),     IonicStorageModule.forRoot({       name: 'mydb',       storeName: 'users',       driverOrder: ['sqlite', 'websql']     })
how will I manage this?

In that case I think you will have to ask from Ionic-storage to also expose the createInstance() method.
PS: I'm not fully aware of the Ionic-storage API

Really?
I thought you guys have collaborated closely to each other.
anyways, thanks man!

Actually it seems that forRoot() leads to the constructions of a new Store object, which leads to a call to the createInstance() method. So, you might be able to use multiple IonicStorageModule.forRoot({}) calls and get the respective store objects.
Hope this helps!

PS: StorageConfig has a storeName property


Add trades snapshot

We don't need to merge this yet. Just wanted to make a start here
Contents so far:

add trades snapshot method. I'll make a corresponding PR to nash-rust too (https://github.com/nash-io/nash-rust/pull/19)
convenience public/private instantiation methods that I needed in the C++ wrapper. I see that there's already work towards a unified interface, so these methods are probably not needed in the future

You're right that I think we do want to unify the interface (and I started a PR for that), but actually want to merge this first as that will be a much bigger change. Merged in https://github.com/nash-io/openlimits/pull/100


Remove duplicate PanResponderExample and update docs

While looking at the pan responder docs I noticed they linked to ResponderExample rather than PanResponderExample and that ResponderExample  defined NavigatorIOSExample which was odd. This PR just kills ResponderExample and updates the link in the docs. :bowtie:

Thanks @jordanbyron, this is an easy merge @vjeux - ResponderExample is indeed unused, not sure how that file got in there but :tada:

@vjeux - this is an easy merge

@facebook-github-bot shipit