JRE error while running westeros_baseline in  jupyter notebook

Problem description
Running westeros_baseline (without any modifications) in jupyter notebook (the Python [conda env:msg_env] kernel) caused error (the report in the log-file)
hs_err_pid43115.log
The error occurs when running cell  #34: scenario.solve()
My best guess is that the message-ix installation is inconsistent with my iMac-env or with the installed GAMS. Unfortunately, I have no idea how to explore and fix the problem.
Versions
message-ix show-versions

ixmp:        3.3.0
message_ix:  3.3.0
message_ix_models: None
message_data: None

click:       8.0.1
dask:        2021.06.0
genno:       installed
graphviz:    None
jpype:       1.2.1
… JVM path:  /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home/jre/lib/jli/libjli.dylib
openpyxl:    3.0.7
pandas:      1.2.4
pint:        0.17
xarray:      0.18.2
yaml:        5.4.1

iam_units:   installed
jupyter:     installed
matplotlib:  3.4.2
plotnine:    None
pyam:        0.11.0

GAMS:        35.2.0

python:      3.9.4 | packaged by conda-forge | (default, May 10 2021, 22:13:15)
[Clang 11.1.0 ]
python-bits: 64
OS:          Darwin
OS-release:  20.5.0
machine:     x86_64
processor:   i386
byteorder:   little
LC_ALL:      None
LANG:        None
LOCALE:      (None, 'UTF-8')

I attach also the output to console; maybe it contains useful info?
console.txt

I run the a copy ipynb Muhammad presented today and got another exception (copied below) which shows that my message-ix installation is faulty. I've checked GAMS (with their transport model): it appears to work OK. Please suggest how to trace the source of the problems (the above and below summarized). My best guess from the follow is that jpype continues to be the trouble-maker.
scen.solve()
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in at.ac.iiasa.ixmp.objects.MsgScenario.toGDX()

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in at.ac.iiasa.ixmp.objects.Scenario.makeGamsDB()

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in com.gams.api.GAMSWorkspace.<init>()

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in com.gams.api.GAMSWorkspace.initializeAttributes()

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in com.gams.api.GAMSWorkspace.verifySystemDirectory()

Exception: Java Exception

The above exception was the direct cause of the following exception:

com.gams.api.GAMSException                Traceback (most recent call last)
<ipython-input-29-e0c31090e7e2> in <module>
----> 1 scen.solve()

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/message_ix/core.py in solve(self, model, solve_options, **kwargs)
    563             :class:`.GAMSModel`.
    564         """
--> 565         super().solve(model=model, solve_options=solve_options, **kwargs)
    566
    567     def add_macro(self, data, scenario=None, check_convergence=True, **kwargs):

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/ixmp/core.py in solve(self, model, callback, cb_kwargs, **model_options)
   1666         # Iterate until convergence
   1667         while True:
-> 1668             model.run(self)
   1669
   1670             # Store an iteration number to help the callback

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/message_ix/models.py in run(self, scenario)
    361
    362         try:
--> 363             result = super().run(scenario)
    364         finally:
    365             # Remove the optfile regardless of whether the run completed

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/ixmp/model/gams.py in run(self, scenario)
    274         try:
    275             # Write model data to file
--> 276             scenario.platform._backend.write_file(
    277                 self.in_file, ItemType.SET | ItemType.PAR, **s_arg
    278             )

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/ixmp/backend/jdbc.py in write_file(self, path, item_type, **kwargs)
    576             # include_var_equ=False -> do not include variables/equations in
    577             # GDX
--> 578             self.jindex[ts].toGDX(str(path.parent), path.name, False)
    579         elif path.suffix == ".csv" and item_type is ItemType.TS:
    580             models = set(filters.pop("model"))

com.gams.api.GAMSException: com.gams.api.GAMSException: GAMS system directory is both not specified and not found from your environment!

@marek-iiasa can you please check if you correctly specified GAMS in your environment variable as mentioned in installation instructions here.

yes, gams, as well as java, are in the $PATH: ```
(msg_env) Ba4:wrksp marek$ echo $PATH
/Users/marek/anaconda3/envs/msg_env/bin:/Library/Frameworks/GAMS.framework/Resources:/Applications/anaconda3/condabin:/anaconda3/bin:.:/usr/local/git/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/Java/Home/bin:/Users/marek/bin:/Users/marek/src/mcma/ScriptsMac:/Library/TeX/texbin:/Library/Apple/usr/bin
I assume the including in the $PATH all one has to do, or ?

I did the install this morning on macOS:
step1. I deleted Anaconda and GAMS.
step2. rebooted machine.
step3. installed GAMS 35.2 (including ticking the box to add to PATH)
step4. installed anaconda3-2021.05
step5. installed messageix via anaconda following each step of the instructions.
step5. tried to download tutorials - the following error occurs

Many thanks, Oliver. Comparing your and my installations shall help me to narrow possible problems with my installation. The key "suspect" I noticed is JVM path: your points to the message_env while mine points to the JDK (which I installed and added to $PATH).
Please comment on:

Have you installed JDK and added its bin to your $PATH ?
How your $JAVA_HOME is defined?
Should JDK be added to $PATH or defining $JAVA_HOME is enough?
Confirm that a tutorial (coffee_machine or westeros) opt-task was solved by gams; the show-versions in my installation didn't show errors; the exception occurred during scen.solve()
There is no plotnine in my env; this might be due to uses of pip (instead of conda) for my installation
I noticed that we have slightly different OS-releases (19.6 vs 20.5) but I guess this unlikely causes problem with my installation.

I will reinstall message-ix when I'll be sure that our environments are as close as possible, i.e. after you will comment on the first four questions.

Marek,
As for questions 1, 2 and 3 -> No, I havent installed JDK.
4. Yes, the Westeros tutorial solved.
5. No, i believe this shouldnt create any issues.
6. Plotnine is only required for some reporting functionalities which are not called during the actual solving.
7. I have an older version of macOS, because the BigSur update caused macs of my type to be "bricked" and since then I have been hesitant to upgrade. But this also shouldnt make any issues.

Thanks again, Oliver,
Aha... Does it mean you have no $JAVA_HOME defined (as required by the installation instructions)?

Marek, no i didnt define $JAVA_HOME. This is only needed if Java is installed manually. I didnt do this so it is done so as part of the MESSAGEix installation process. So when calling which java, i get:
/Users/oliver/opt/anaconda3/envs/message_env/bin/java

Oliver, thanks again. I'll reinstall message_ix later today following your path, and report result here.

Hi @marek-iiasa, is this issue still relevant? If not, we can close it to have a better overview of what issues might require our attention :)

I cannot contribute here as I have no idea about this issue

no comments


Truncated contacts sidebar

Goofy's feature, might be a nice enhancement?

This project aims to bring Messenger to the desktop, not reinvent it or derive from it. I believe in the Facebook Messenger team and for them to make sensible UI decisions — if messenger.com adds this feature we will see it here, if not: not.

Totally.


Add Java callable factory generation

Corresponding gax-java PR: https://github.com/googleapis/gax-java/pull/483

Codecov Report

Merging #1855 into master will increase coverage by 0.03%.
The diff coverage is 98.07%.

@@             Coverage Diff              @@
##             master    #1855      +/-   ##
============================================
+ Coverage     84.77%   84.81%   +0.03%
- Complexity     4635     4642       +7
============================================
  Files           438      439       +1
  Lines         18384    18435      +51
  Branches       1947     1948       +1
============================================
+ Hits          15585    15635      +50
  Misses         2128     2128
- Partials        671      672       +1

Impacted Files
Coverage Δ
Complexity Δ

.../transformer/java/JavaGapicSurfaceTransformer.java
100% <ø> (ø)
6 <0> (ø)
:arrow_down:

...sformer/java/JavaDiscoGapicSurfaceTransformer.java
93.75% <ø> (ø)
6 <0> (ø)
:arrow_down:

...e/api/codegen/viewmodel/StaticLangRpcStubView.java
100% <ø> (ø)
2 <0> (ø)
:arrow_down:

...m/google/api/codegen/transformer/SurfaceNamer.java
72.44% <100%> (+0.07%)
204 <1> (+1)
:arrow_up:

...degen/viewmodel/StaticLangCallableFactoryView.java
100% <100%> (ø)
2 <2> (?)

...degen/transformer/java/JavaSurfaceTransformer.java
97.67% <97.91%> (+0.01%)
75 <4> (+4)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 118d18d...8c579ae. Read the comment docs.

PTAL


add activeClassName to tab component

Addressing #294.

Nice, thanks!

@javivelasco
Glad these were able to get merged in!  Do you have any idea as to your next release?

Next weekend!


[Service Bus] Handle cases where desired worker count exceeds int.MaxInt

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

API change check
API changes are not detected in this pull request.

azp run net - servicebus - tests

/azp run net - servicebus - tests

/check-enforcer override

Failures appear to be transient and unrelated to the changes made.   Merging.


feat: add textarea component

ref #5

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

3 of 5 (60.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.4%) to 62.069%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/fuck/textarea/index.tsx
2
4
50.0%

Totals

Change from base Build<PHONE_NUMBER>:
-0.4%

Covered Lines:
18

Relevant Lines:
29

💛  - Coveralls


Svg previews should be seen on thumbnail view

Svg previews is a good idea but nobody uses preview window.
SVG icons should be seen on explorer icons.
Html previews should also be included (I remember this useful feature existed on previous versions of windows but it is not available on windows 10)

tracking as #1341

Thanks for the suggestion too!


plugin accessing local chroma persistent instance

Hi TaskWeaver Team,
I'm building a plugin for RAG search with Chroma.
When I tested the plugin Python file standalone, the plugin successfully retrieved data from the Chroma persistent client. However, when I asked a question in the TaskWeaver UI that needed a RAG search, the code interpreter triggered the plugin, but the plugin couldn't retrieve data from the Chroma persistent client, saying "no relevant information."
I was wondering if there is any limitation on plugins accessing local directories or the persistent client.
Thanks!

Could you check the generated code? Is that as expected? You can paste more logs here for analysis.

Hi liqul, I've found the issue. I got multiple plugins using the same chroma persistent client which caused some conflicts. now it's fixed. Thanks!


fix: adding --force to rm cmd for suppressing missing operand exception

:tada: This PR is included in version 1.4.6 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


fix: support getting table properties using schema with db name prefix

With the introduction of PR #835 a user might want to fully specify a schema with the database name as prefix, i.e. "my db"."my schema".
This PR fixes a bug that currently prevents usage of Dialect.has_table, Dialect.get_table_names, Dialect.get_columns, Dialect.get_foreign_keys and Dialect.get_check_constraints when using said fully specified schema name.
I also refactored test_get_schema_names so there is a separate test for get_table_names.

@Mause, this is now ready for review. I added a method _pg_class_filter_scope_schema which filters the query by the specified schema. This only fixes the original bug partially, however - we don't cover the edge case where a schema + table with the same name exists in another database (see note on line 488). I suggest to address this in a subsequent PR.

Please review/merge this PR to address this issue: https://github.com/apache/superset/issues/26286


chore: run semantic-release only once

Inspired by https://github.com/semantic-release/semantic-release/blob/caribou/docs/recipes/travis-build-stages.md

:tada: This PR is included in version 5.1.0 :tada:
The release is available on:

npm package (@next dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Remove bluebird/core-js

Remove unneeded types:

bluebird
core-js

Reported by @erickoledadevrel

Removed.


#23  update arduino file and android app to new broker connection

The arduino files has been improved and updated.

Added a new structure
Created new topics
Added the set up for the accelerations buttons that will come in the future.
New values for the executions of the car movement

Android app:

The name of the app has been changed
Creation of enum class and a movement connection class to avoid redudancy in the implementation of the joystick
New topics has been created
The controlPad class has been updated to work with the new structure

Related issue: #23

Why are you mentioning the issue in the title as issue 23 instead of the standard and traceable way, ie #23?

I did use  #23 in the first push comment and the pull request is also connected to the issue.
When I created the branch to an issue, GitHub suggested that name to the issue branch so I tried it out.
Skickat från min iPhone

28 apr. 2022 kl. 12:42 skrev Dimitris Platis @.***>:

Why are you mentioning the issue in the title as issue 23 instead of the standard and traceable way, ie #23?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were assigned.

You can call the branch something that implies the issue since there's no other way to refer to it there, but in a pull request there are standardized ways of doing so (ie referring to issues) instead.

thank you for the feedback but Im wondering if I understand you right.
We need to refer the issue in three parts then? because know I have use it (1) in the  comment and also (2) in the right column of the pull request. But I should also use it in the pull request name? So it need to be mentioned in three parts.

You must to have the issue (1) linked to the pull request (i.e. "in the right column") and (2) it's really good if you can refer to it in the description of the pull request as long as the issue should be closed after the pull request is merged.
Regarding referring to it in the title of the pull request, I'd say that's the least important and I personally wouldn't do it, but if you want to, make sure you do it right and add the # before the number of the issue.


Add getBestMove and FirstBestAdmissibleTabuSearch

Add a new getBestMove to NeighbourhoodSearch() with a parameter boolean acceptFirstImprovement.
Add FirstBestAdmissibleTabuSearch.java

Coverage decreased (-2.08%) to 95.095% when pulling 1c9e3b8b3d6c0cb9d517f4dd3d5564ec23cad7c7 on huanfachen:tabuSearch/add_variant into f6bfd50ac978d1d392ab6bf8cefccfad712ff077 on hdbeukel:master.

Coverage decreased (-2.1%) to 95.095% when pulling e06489bed6a6ec09d32731bc1049efd71d458f4d on huanfachen:tabuSearch/add_variant into f6bfd50ac978d1d392ab6bf8cefccfad712ff077 on hdbeukel:tabuvariant.

Coverage decreased (-0.7%) to 96.555% when pulling a25b183a3878f1cfb8e37c56e5f24a252bcf477e on huanfachen:tabuSearch/add_variant into f6bfd50ac978d1d392ab6bf8cefccfad712ff077 on hdbeukel:tabuvariant.

Coverage decreased (-0.7%) to 96.555% when pulling df1fe558d3f873e5d33dc35cd4c83a3a4c90bf4a on huanfachen:tabuSearch/add_variant into f6bfd50ac978d1d392ab6bf8cefccfad712ff077 on hdbeukel:tabuvariant.

Please remove the following files and folders (IntelliJ/Eclipse settings) and include them in .gitignore to avoid that they are added in the future:

.classpath
.project
.settings
james-core.iml (add *.iml to .gitignore)
.idea

Also see my other comments.

You have several compiler warnings in your code. See https://travis-ci.org/hdbeukel/james-core/builds/238702467?utm_source=github_status&utm_medium=notification. Please fix.

The undesired files listed above are also still present.

Coverage decreased (-0.08%) to 97.146% when pulling 5d564cb959156a4200050f7fff1ce0b9efbdf685 on huanfachen:tabuSearch/add_variant into f6bfd50ac978d1d392ab6bf8cefccfad712ff077 on hdbeukel:tabuvariant.

Coverage decreased (-0.1%) to 97.096% when pulling 9eaf5c4ffd50f579c196f6b6c475742d5793c3ee on huanfachen:tabuSearch/add_variant into f6bfd50ac978d1d392ab6bf8cefccfad712ff077 on hdbeukel:tabuvariant.

Tests fail now because you have deleted the file pom.xml (alongside the undesired files).

Coverage decreased (-0.08%) to 97.146% when pulling 37e899e9798ea431d2db346399184541594ea816 on huanfachen:tabuSearch/add_variant into f6bfd50ac978d1d392ab6bf8cefccfad712ff077 on hdbeukel:tabuvariant.

Thanks for the fix. I will get back to this next week and merge your pull request (no time this week).

I will probably make some more changes (e.g. Javadocs).  I will send you a pull request.


Title and Hamburger icon are hiding when reducing the height of the Navigation Bar

[ ] Issue does not already exist
Identify exactly what aspect your issue is about

[x] ignite base application
[ ] ignite the cli
[ ] ignite generated code

Sorry, I followed the below one but I couldn't figure out what I am missing. When I reduce the height of the Navigation Bar, Title & Icons are hiding.
https://github.com/infinitered/ignite/issues/154
Also, I would like to have the same background image on the Navigation Bar but by default, I can only change the colors. Is there any option to keep the same BG on Navigation Bar & Drawer?
Thank you

Please close this issue. I made the below changes to navButtonLeft in App/Navigation/NavigationStyle.js
Replaced this line, "margin: Metrics.baseMargin" with
marginLeft: Metrics.baseMargin,
marginRight: Metrics.baseMargin

It solved my problem. Thank you!


ETag usage

I cannot find any usage of etag fields or headers in the codebase, but for my understanding this should be used.
E.g.

for actors reminders, see dapr/dapr#677 and dapr/docs#211
for state, see https://github.com/dapr/docs/blob/master/reference/api/state.md

As etag will not be exposed for Actors as runtime ensures their single threadedness, it's still the question if we need them to be used with state management, e.g. expose them through the SDK?

Yes, for state management it should be exposed.

I can do this, if you've not already started

Sure, thanks, please assign it to yourself.

Tracked by P0 item https://github.com/dapr/dotnet-sdk/issues/225
This item will add state apis as wrapper over gRPC api.


Repos file size mismatch

Describe the issue
Looking at the metadata https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/InRelease , then
https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/main/binary-amd64/Packages.gz: size check mismatch 313493 != 313128
The same goes with metadata from https://packages.microsoft.com/repos/edge/dists/stable/InRelease and then
https://packages.microsoft.com/repos/edge/dists/stable/main/binary-amd64/Packages.gz: size check mismatch 11350 != 11210
And https://packages.microsoft.com/repos/vscode/dists/stable/InRelease and
https://packages.microsoft.com/repos/vscode/dists/stable/main/binary-amd64/Packages.gz: size check mismatch 26915 != 26670

When did the issue occur?
For at least since Sept 12th until creation of this issue
If applicable, what package did you attempt to install, and from which repo?
Steps to Reproduce

Check file size documented in InRelease, and compare to actual file size
Actual Result
Size don't match
Expected Result
Size Match
Screenshots

Additional context

@fadnincx it's possible that these files will be out of sync. We are considering lowering the TTL so the window will be smaller but really you should be using the by-hash files which we provide (e.g. https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/main/binary-amd64/by-hash/). I am curious why are you pulling the Packages.gz instead of the by-hash files?

For one, I'm trying to mirror the repos with Aptly (the same software apparently used to publish the repos (at least according to the description) so my influence on that files of a repo are used how is quite limited and second of all, it doesn't help, if one gets a 404 when trying to access the resources. In my opinion, older versions should stay online at least for some time. But the by-hash file contains only the most recent version.
Digging a bit deeper, it seems that the 'InRelease' file containing the hashes, file size and filenames, already has a caching issue.
When using curl on https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/InRelease with explicitly asking for no cache:
*   Trying <IP_ADDRESS>:443...
* Connected to packages.microsoft.com (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=packages.microsoft.com
*  start date: Sep 11 14:10:52 2024 GMT
*  expire date: Sep  6 14:10:52 2025 GMT
*  subjectAltName: host "packages.microsoft.com" matched cert's "packages.microsoft.com"
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure RSA TLS Issuing CA 04
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55b42395eb20)
> GET /ubuntu/20.04/prod/dists/focal/InRelease?1726516400 HTTP/2
> Host: packages.microsoft.com
> user-agent: curl/7.74.0
> accept: */*
> cache-control: no-cache
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Mon, 16 Sep 2024 19:53:20 GMT
< content-type: application/octet-stream
< content-length: 3632
< last-modified: Wed, 11 Sep 2024 08:40:03 GMT
< etag: "0x8DCD23D54D4A1FD"
< x-ms-request-id: 6da57028-f01e-0000-5727-04a06d000000
< x-ms-version: 2024-08-04
< x-ms-version-id: 2024-09-11T08:40:03.7827069Z
< x-ms-is-current-version: true
< x-ms-creation-time: Wed, 11 Sep 2024 08:40:03 GMT
< x-ms-blob-content-md5: ny6TOkS08eaJ/8QsqHam+w==
< x-ms-lease-status: unlocked
< x-ms-lease-state: available
< x-ms-blob-type: BlockBlob
< content-disposition: attachment;filename=InRelease
< x-ms-server-encrypted: true
< strict-transport-security: max-age=31536000; includeSubDomains
< x-content-type-options: nosniff
< x-azure-ref: 20240916T195320Z-1794b988df8kqgr7wy89a4assg00000006k0000000008fn1
< x-cache: TCP_HIT
< cache-control: public, max-age=300
< x-fd-int-roxy-purgeid: 66949960
< accept-ranges: bytes
<
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Origin: microsoft-ubuntu-focal-prod focal
Label: microsoft-ubuntu-focal-prod focal
Suite: focal
Codename: focal
Date: Wed, 11 Sep 2024 08:34:49 +0000.
Architectures: amd64 arm64 armhf all
Components: main
Description: Generated by aptly
Acquire-By-Hash: yes
....

while opening the same in firefox gives:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Origin: microsoft-ubuntu-focal-prod focal
Label: microsoft-ubuntu-focal-prod focal
Suite: focal
Codename: focal
Date: Mon, 16 Sep 2024 16:53:36 +0000
Architectures: amd64 arm64 armhf all
Components: main
Description: Generated by aptly
Acquire-By-Hash: yes
...

Digging into the http header, the difference seems to be with Accept-Encoding: gzip or Accept-Encoding: br. So definitely something on the cache-level is wrong

We did generate some of the repos at one time using aptly but we no longer use it. I assume that aptly is not using by-hash to mirror repos?

In my opinion, older versions should stay online at least for some time. But the by-hash file contains only the most recent version.

They should remain available from 3 days for up to 1 year. If you are using the html directory listing, then that's only showing the files in the latest version of the repo. It doesn't show the cached files including the cached by-hash files but they should still be available to download even though they aren't listed.
I'm also seeing issues with different versions of the InRelease file being returned so I've asked the CDN team to investigate.

This exact scenario is impacting me. While it is "only" one repo, when relying on the replicated copy (we user https://github.com/apt-mirror/apt-mirror/blob/master/apt-mirror), it prevents our entire automation (puppet) from running.
Can you please regenerate the Packages.gz with the "correct" values (well, at least to let 'apt update' run).

We did generate some of the repos at one time using aptly but we no longer use it. I assume that aptly is not using by-hash to mirror repos?

I can neither confirm nor deny that aptly uses by-hash or not, because it might fall back to files if the hash files are not available.

In my opinion, older versions should stay online at least for some time. But the by-hash file contains only the most recent version.

They should remain available from 3 days for up to 1 year. If you are using the html directory listing, then that's only showing the files in the latest version of the repo. It doesn't show the cached files including the cached by-hash files but they should still be available to download even though they aren't listed.

I have not yet found any files via hash that are not latest and available. So no, it does not seem that they are just excluded from the html file listing, but that they are not available at all.
Picking any hash from the InRelease version 9/11/24 currently delivered to curl give me a 404 error, such as for example (taking the first 3 hashes)

https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/main/binary-amd64/by-hash/SHA256/adf7a01031efa32e2b8ecef4608f8c17e496df978f2bcb983a9baec3d90599e0
https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/main/binary-amd64/by-hash/SHA256/c69501ca49ce1174fb08347ce2e1e22a1e5cb0347da17fc039c60f6680a1962a
https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/main/binary-amd64/by-hash/SHA256/b698a175c9b594ff561ae2e5cfc2a3a51a79f3f1afe9a1adc0aacb5f3f74867d

The by-hash files are stored in cache for at least three days after they are removed from the repo. These were removed on September 12 so they probably became unavailable sometime after September 15 but before today. We can increase that but right now I think the bigger problem is figuring out why old versions of InRelease are still being served.

Just chiming in that we're seeing this too on some of our machines running in eu-central-1. Is there a workaround that we can apply while this is being resolved?
Reading package lists...
E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/main/binary-arm64/Packages.gz  File has unexpected size (43334 != 43043). Mirror sync in progress? [IP: <IP_ADDRESS> 443]
   Hashes of expected file:
    - Filesize:43043 [weak]
    - SHA512:88b58c2d6a0cebf09[63]
   Release file created at: Thu, 12 Sep 2024 18:17:22 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

We think that during our infrastructure deployment last week, apt metadata files (InRelease, Packages.gz, etc) became stored in our CDN cache with longer than expected TTLs. I've gone through and purged all apt metadata files from our cache, which should fix the problem. Would be interested to know if anyone is still experiencing issues.

Re-pulled repo today - all is well now.

Now getting the same problem with the microsoft-edge-stable repo..

Re-pulled the edge repo again - all is well now.  Thanks.

All good for me. Curious to see if the cache TTL issue is solved permanently or just for now, because old versions have been manually purged.


fix: proposal json max width and scroll

Closes: #100
This pull request adds max width and scroll to the UI card that displays messages in json format.
Preview:

https://deploy-preview-114--regen-groups-ui.netlify.app

Example:

https://deploy-preview-114--regen-groups-ui.netlify.app/37/proposals/60

Looks a bit weird on mobile. Not blocking since we don't actually support wallet connect anyway for now. But probably good to keep in mind.


Improve placement of double-daggers

Remove the double-daggers where we really do mean statements not steps (e.g., in some areas of the bottom-up prover).
Add double-daggers to the use of "substitution"; my hope is that the tool will show "replacement" everywhere eventually. That way we can use the word "substitution" for the underlying Metamath substitution operation, and "repacement" for the editor menu item. It's always nice to consistently use two different words for two different things :-).

Note: This is related to PR https://github.com/expln/metamath-lamp/pull/102 on metamath-lamp itself. That PR changes the icon's term "substitution" to "replacement", which I think will reduce confusion.

The double-daggers are a temporary measure. My intent is to replace the double-daggers with the new words when v11 comes out. That's certainly true fit statement; for changing "substitute" to "replace" that depends on an unmerged PR.


Unit Tests for Tools

PHPUnit tests should be written to cover the Tools section of the site.

A user should be able to visit the Tools page
A user should be able to click Enable/Disable Maintenance Mode, and the Artisan Commands up or down should be run, and the page reloads back to the Tools page with a success message
A user should be able to click Download Archive and should have downloaded an Archive folder to storage/, it downloads, and then the folder and zip are deleted from storage/
A user should be able to click Reset Index, and the Artisan Command canvas:index should be run, and the page reloads back to the Tools page with a success message
A user should be able to click Clear Cache, and the Artisan Commands cache:clear, route:clear and optimize should be run, and the page reloads back to the Tools page with a success message

@RDelorier Thought you might take a stab at this one, as you're a testing ninja ;)

Fixed with 52dad2771879980057896d14fd8c3a68de3eb224


dead characters aren't displayed properly

Seems like we need a GTK IMContext for that, see https://gitlab.gnome.org/GNOME/gtk/blob/gtk-3-24/gtk/gtkentry.c#L2811. This might also fix #169 while we're at it.


Feature request: Listing enabled addons

It would be nice if there were an easier way to query k8s to find out what plugins are enabled. Something like microk8s.enabled --list could expose the list in a way that's consistent with the current command, I think.
You can poke around kubectl get all -n kube-system but it's not a 1:1 map between what we enable explicitly and what's on by default.
For my use-case, I want to know what addons are enabled so that I can make an operational decision. In other words, if dns is already enabled, I don't want to enable it a second time because it goes through the process (applying manifest/creating services/etc and restarting kubelet.

If I may add, a machine friendly response output would also help in integration with external systems and automating tasks (eg microk8s.status --format yaml).
A rest server to expose all management ops paired with a good looking UI would appeal to both integrators and users new to Kubernetes.


Programmatically Dismiss View

Hello. Great job on this framework!
Is there a way to programmatically dismiss the JSAlertView? I am trying to have it dismiss from a shake gesture.
Thank you.

Was able to solve this on my by adding this to the JSAlertView class:
// Check for shake gesture, vibrate, then remove alert view
override public func motionEnded(motion: UIEventSubtype, withEvent event: UIEvent?) {
if(event!.subtype == UIEventSubtype.MotionShake) {
closeView(true, source: .Close);
}
}

This is partial solution, because if I implement this it will be active always even for those who do not want to have this functionality in their app.

I will do it as soon as I will find a time for it.

You can always implement your solution inside the viewcontroller JSSAlertView is called from.
Keeping it like this.

Why closeView method isn't public?

Good question.
It was not intended to be used in the first iteration.
I actually do not see any reason why it should not be public now.

Thanks, for fast response! That would be great!


Minor updates due to move to Apache

I noticed some places used fluo and other uses incubator-fluo in the links.

Merged in 83eba45dbec449a84cd65de645767cf03f843d3f


Consider allowing packages to define multiple canonical names for their modules

Suggestion
This is based on a change in the NodeJS definitions: https://github.com/DefinitelyTyped/DefinitelyTyped/pull/51107. For Node's ESM support, you can import NodeJS core packages (i.e., fs, path, etc.) using a fully-qualified URN with the node: prefix. The approach taken in https://github.com/DefinitelyTyped/DefinitelyTyped/pull/51107 is to re-export a module like fs using another ambient module:
declare module "node:fs" {
  export * from "fs";
}
declare module "fs" {
 ...
}

However, this belies the actual behavior of the import. "node:fs" is not a different module that happens to re-export "fs", it is actually the same module, and ideally would be handled in a way similar to how we currently handle symbolic links.  With the above definitions, any module augmentations must always be made against the non-prefixed form of the module. Augmentations of the prefixed-form would be missing when importing the non-prefixed form.
We already have some mechanisms for handling this in your own project (i.e., via "paths", etc.), but not when importing from an external package (such as is the case with "fs", "path", etc.).
Should we consider adding a mechanism, be it either via syntax or via configuration (stored in package.json)?
Syntax
We can bikeshed on syntax, but I've presented a few options below:
// 1) variation of `export as namespace ...`
declare module "fs" {
  export as module "node:fs"; // essentially a symbolic link from `"node:fs"` to `"fs"`
}

// 2) declare a module with multiple names
declare module "fs", "node:fs" {
}

Configuration
If we consider approaching this problem from a configuration standpoint there are a number of directions we could take. I can update this section as we discuss this issue.
🔍 Search Terms
NodeJS, ESM, prefix, symlink

Also should consider which module name takes precidence when automatically importing, probably want to use declaration order if  we go with the declare module "fs", "node:fs"  syntax, or have some sort of override.

As a side note regarding the node: protocol, looks like the node: scheme / prefix was backported to Node.js 14.13.1 / 12.20.0 🎉
https://nodejs.org/api/esm.html -> link "node: Imports" (links with anchors don't scroll properly right now)

<EMAIL_ADDRESS>has been published now - the node: schema / prefix now works again:

https://www.npmjs.com/package/@types/node
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/53669


Reloading the page does not show changes

What version of Hugo are you using (hugo version)?
$ hugo version
hugo v0.134.1-2f89169baa87a9db47e288b60705f4e99e21a945+extended windows/amd64 BuildDate=2024-09-05T10:17:50Z VendorInfo=gohugoio

I am using Windows 10.
Does this issue reproduce with the latest release?
Clone and setup my blog.
git clone https://github.com/jsmolka/blog
cd blog
npm i
hugo server -F -D

Then change the content of any markdown file. Hugo detects the change, but it's not visible in the browser. Even reloading the page doesn't reflect the changes. I doesn't work with Chrome and Firefox. I need to restart hugo to see the changes. I had this issue for a while and stayed on a older hugo version, hoping for the fix. It stopped working when upgrading from v0.122.0 to v0.123.0.
Hope this helps, thanks for the otherwise great library.

This mount is causing the problem:
[[module.mounts]]
source = "content"
target = "assets"

If you remove the above from your site configuration, the rebuild works as expected.
I understand what you're trying to do (make every page resource available as a global resource), but I wouldn't do that.

Failing test:
func TestRebuildIssue12838(t *testing.T) {
	t.Parallel()

	files := `
-- hugo.toml --
disableKinds = ['page','section','rss','sitemap','taxonomy','term']
[[module.mounts]]
source = 'content'
target = 'assets'
-- layouts/index.html --
{{ .Content }}
-- content/_index.md --
foo
	`

	b := hugolib.NewIntegrationTestBuilder(
		hugolib.IntegrationTestConfig{
			T:           t,
			TxtarString: files,
			NeedsOsFS:   true,
			Running:     true,
		}).Build()

	b.AssertFileContent("public/index.html", "foo")

	b.EditFiles("content/_index.md", "bar").Build()

	b.AssertFileContent("public/index.html", "bar")
}

Thank you. I can't really remember what I was trying to achieve there, but I removed the mount and everything works again.


Error in tf.keras.metrics.AUC

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS Catalina
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No
TensorFlow installed from (source or binary): No
TensorFlow version (use command below): 2.4.1
Python version: 3.6.6
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with:

TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior
tf.keras.metrics.AUC(from_logits=True) and tf.keras.metrics.AUC(from_logits=False) outputs TypeError: __init__() got an unexpected keyword argument 'from_logits'
Describe the expected behavior
Contributing - Do you
want to contribute a PR? (yes/no): - Briefly describe your candidate solution
(if contributing):
Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.
Reproducible error: https://colab.research.google.com/drive/1-ZMVqG7t54a2QO-Vh2zKDdyT2gKYB47u?usp=sharing
Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __init__() got an unexpected keyword argument 'from_logits'

Cause It is available from TF 2.5.0
Please upgrade your TF version

p.s. in Colab pip install --upgrade tensorflow

This is also mentioned in TF 2.5 release notes. See https://github.com/tensorflow/tensorflow/releases

Thanks!


Wait... Why is the directory structure different in develope?

In addition, my PR was based on the main branche, so it should actually be merged there as well

They will be removed in a short while and also i will be change the language files according to your corrections. But they are not exists in develop with yml format.


Update instructions for self-hosting Shields with Heroku

The Heroku setup instructions were outdated. There's no need for a build pack.

Messages

:book:
:sparkles: Thanks for your contribution to Shields, @henrypoydar!

:book:

Thanks for contributing to our documentation. We :heart: our documentarians!

  Generated by :no_entry_sign: dangerJS against db93b46eae9c5257469c149ca780b7660f6ff101

Thanks @henrypoydar (and good timing! --> https://github.com/badges/shields/pull/3115)


Create export-amr-api-permission-guide

Article describing the asynchronous metadata read api  permission guidance.
Category

[X ] New article

Related issues
Will be updating the TOC to reference this.
What's in this Pull Request?
New article description the AMR API permission options.

@JoanneHendrickson please correct issues with this PR...

the file has no extension... appears it should be markdown
the file is placed in the root of the site... shouldn't it be in a subfolder in the docs somewhere?

@andrewconnell   Yes, thank you Andrew.  The revised description works fine.


Subset used in MDX query

Describe what did you try to do with TM1py
Does anyone know if you can assign a subset to the rows in an MDX query?  I am trying to use a subset I created on a new MDX pull but am not sure how to do this.  Also, does anyone know if you can use the roll-up of the subset in the MDX query (this would be preferred)?
Describe what's not working the way you expect
Didn't get the expected result? Describe:
I have a script creating a subset by subtracting elements that have data in 1 or both of 2 cubes from all elements in the dimension, leaving those with no data, code is below.  I want to then create a new MDX query against just this subset, or even the rolled-up subset so I can drill down to all accounts.  I cannot do this initially because I run out of memory.
df2 = tm1.cubes.cells.execute_mdx_dataframe(mdx2)
for _, row in df2.iterrows():
    material = row['Material EMEA']
    used_assumption_materials.add(material)

unused_materials = set(all_materials) - used_staging_materials - used_assumption_materials

dimension_name = "Material EMEA"
subset_name = "Unused Materials EMEA"
elements = set(unused_materials)

s = Subset(dimension_name=dimension_name, subset_name=subset_name, alias='', elements=elements)
tm1.dimensions.subsets.create(subset=s, private=False)

Preferably in the next query (below) would like to use the roll-up of the subset created for the Materials and then just drill down on all N level accounts.  I am using all the accounts that had data when I checked this because i don't know how to use a roll-up or the subset.  I want to remove the elements from the first set that have data at N level accounts that gets reversed. and total to 0 at Total Account.
mdx = """
   SELECT
   NON EMPTY {{TM1FILTERBYLEVEL( TM1SUBSETALL( [Material EMEA] ), 0)}} *
   {{[GL Account].[40100000],[GL Account].[40101500],[GL Account].[50156000],
   [GL Account].[40200000],[GL Account].[40120000],[GL Account].[40850000],[GL Account].[40880000],
   [GL Account].[40160000],[GL Account].[50100000],[GL Account].[50199300],[GL Account].[50200000],
   [GL Account].[50120000],[GL Account].[50810000],[GL Account].[50860000],[GL Account].[50880000]}} ON ROWS,

Version

TM1py  - 1.4.1
TM1 Server Version: <IP_ADDRESS>

Additional context
Thanks

Hi @jbeanz23
IMO, if you have the list of elements filtered better use them directly in the MDX instead creating a subset with them.
For the first part of your question: You can use the subset name just like any other element in a dimension in the MDX query, you can also use the TM1SUBSETTOSET function  {{TM1SUBSETTOSET([Material EMEA], {subset_name})}}.
For the second part of your question: I'm not sure if this can be achieved with REST based MDX requests. I know it works in Architect or Perspectives.
Instead of using iterrows you can call unique on a series to get the unique elements (much faster).
df2 = tm1.cubes.cells.execute_mdx_dataframe(mdx2)
used_assumption_materials = set(df2['Material EMEA'].unique())

unused_materials = set(all_materials) - used_staging_materials - used_assumption_materials

dimension_name = "Material EMEA"
subset_name = "Unused Materials EMEA"
elements = set(unused_materials)

s = Subset(dimension_name=dimension_name, subset_name=subset_name, alias='', elements=elements)
tm1.dimensions.subsets.create(subset=s, private=False)

mdx = f"""
   SELECT
   NON EMPTY {{[Material EMEA].[{subset_name}]}} *
   {{[GL Account].[40100000],[GL Account].[40101500],[GL Account].[50156000],
   [GL Account].[40200000],[GL Account].[40120000],[GL Account].[40850000],[GL Account].[40880000],
   [GL Account].[40160000],[GL Account].[50100000],[GL Account].[50199300],[GL Account].[50200000],
   [GL Account].[50120000],[GL Account].[50810000],[GL Account].[50860000],[GL Account].[50880000]}} ON ROWS,

@jbeanz23 @rkvinoth
Rather than issuing multiple queries, finding the common (or dissimilar) members and then querying again, I would suggest doing the filtering directly in MDX. The query will be "longer" but you can rely on server side optimizations to improve the speed. Once that query works, you can add a dynamic total via a calculated member.
From what I understand, you have a query that returns a set of members on rows, you then want to exclude those members from the the subsequent query. The query would take this format.
SELECT NON EMPTY EXCEPT( {TM1SUBSETALL([Material EMEA])},  {FILTER( {TM1SUBSETALL([Material EMEA])}, [Cube 1].([Dim1].[H1].[M1], [Dim2].[H2].[M1]) <>0)})
The row set leverage both except and filter to get the elements you want.
On the memory issue, is TM1 giving you a view memory error, or is python running out of RAM?

Hey @rclapp
I guess TM1 because in Pulse I see Memory Pool exceeded and Tm1py says something similar.
The problem is I am needing to query from 2 different cubes that have a common dimension that has 1.9 million elements.  I need to find the unused elements, meaning no data in  either cube for any period.  The primary cube is a transaction cube which has multiple dimensions with over 1 millions elements.  The 2nd cube has less dimensions but also includes huge dimensions.  I need every N level element from Time dimension, due to reversals, which is 3 years of periods.  For the primary cube I also need all N level Accounts due to reversals.   I need to drill down into individual months and accounts against 1.9 million N level materials.
If I use the subset (which is 1.3 million with no data at Total Account) I cannot drill down to account, because I run out of memory with 40 periods and all N level accounts.  Is this achievable using some type of filter and exception statement above?
@rkvinoth - thanks for the tip about iterating.  That will save me lots of time.

This looks like a hardcore exercise. Since you are only worried about the Time and Material EMEA dimensions, lookup all the other dimensions at top-level and take Time dimension to the column.
Send concurrent requests to the TM1 server by splitting the subset elements
Something like below should help you.. just pass the mdx_list to a ThreadPoolExecutor. Handle the MDX in a separate function.
from concurrent.futures import ThreadPoolExecutor
from TM1py.Services import TM1Service

dim_name = 'Material EMEA'
element_list = ['item1', 'item2', 'item3', 'item4', 'item5', 'item6', 'item7']

elements = ["[Material EMEA].[" + item + "]" for item in element_list]
mdx_list = []
subset_size = 3
counter = 0
thread_count = 20

for n in range(subset_size, len(elements) + 1, subset_size):
    mdx = f"""
       SELECT
       NON EMPTY {{[Material EMEA].[{','.join(elements[counter:n])}]}} *
       {{TM1FILTERBYLEVEL( {{TM1SUBSETALL( [GL Account] )}}, 0)}} ON ROWS,"""
    mdx_list.append(mdx)
    counter = n

if len(elements) - counter:
    mdx = f"""
       SELECT
       NON EMPTY {{[Material EMEA].[{','.join(elements[counter:len(elements)])}]}} *
       {{TM1FILTERBYLEVEL( {{TM1SUBSETALL( [GL Account] )}}, 0)}} ON ROWS,"""
    mdx_list.append(mdx)

with ThreadPoolExecutor(max_workers=thread_count) as executor:
            executor.map(function_name, mdx_list)

@jbeanz23
Since your initial question is answered I will close this issue. Feel free to re-open if necessary

While executing below MDX TM1 instance is crashing because of hefty amount of cell combinations . so i split the dept count  based on attribute condition  shown below  .but mdx is not taking these list items . any one of  the below mentioned points solution is ok for me
1.How to pass list items into MDX
2.How to define a specific subset in MDX query
3. How to extract leaf elements based on Parent Element input
######MDX for Filtering dept with attribute###########
mdx="""
SELECT
NON EMPTY {TM1FILTERBYLEVEL( {TM1SUBSETALL([Department])},0)} ON ROWS,
NON EMPTY {[}ElementAttributes_Department].[RD_D]} ON COLUMNS
FROM [}ElementAttributes_Department] """
DF_Attr=tm1.cubes.cells.execute_mdx_dataframe(mdx)
Dept_List=DF_Attr['Department'].to_list()
#################
mdx_list=[]
for Dept in Dept_List:
mdx="""
SELECT
NON EMPTY {TM1FILTERBYLEVEL( {TM1SUBSETALL([Level Type])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Shift Type])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Level Class])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Skills])},0)}
{TM1FILTERBYLEVEL({TM1SUBSETALL([Location])},0)}
{TM1FILTERBYLEVEL({TM1SUBSETALL([Location Type])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Level])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Month])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Resource Unit])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Role])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Vendor])},0)}*
         ON ROWS,
          NON EMPTY {[Measure HC FTE].[FTE (PlanYear)]} ON COLUMNS
          FROM [Input_HC_FTE_for_Externals]
          WHERE ([Version].[Version 1],[Scenario].[1EA],[Year].[2021],[Department].[{Dept}])"""

Wim Gielis has this wonderful page for learning/working with MDX statements, I suggest that you go through it.
For your first question - Check this reply from me
For your second question - Check this reply from me
For your third question - TM1FilterByLevel( Descendants( [<Dimension name>].[<Parent element>]), 0)

I have  executed the  MDX query in threads but i am not getting data output  file  .below is the script
from TM1py.Services import TM1Service
from TM1py.Utils import Utils
import pandas as pd
from concurrent.futures import ThreadPoolExecutor
import re
tm1=TM1Service(address='************',port=22222,user='abcd',password='ccccc',ssl=True,namespace='Active Directory')
mdx="""
SELECT
NON EMPTY {TM1FILTERBYLEVEL( {TM1SUBSETALL([Level Type])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Shift Type])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Level Class])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Skills])},0)}
{TM1FILTERBYLEVEL({TM1SUBSETALL([Location])},0)}
{TM1FILTERBYLEVEL({TM1SUBSETALL([Location Type])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Level])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Month])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Resource Unit])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Role])},0)}*
{TM1FILTERBYLEVEL({TM1SUBSETALL([Vendor])},0)}*
{TM1FilterByLevel( Descendants( [Department].[1100]), 0)}
ON ROWS,
NON EMPTY {[Measure HC FTE].[FTE (PlanYear)]} ON COLUMNS
FROM [Input_HC_FTE_for_Externals]
WHERE ([Version].[Version 1],[Scenario].[1EA],[Year].[2021])"""
def export_data(expression):
data=tm1.cubes.cells.execute_mdx_datafrmae(expression)
data.to_csv('C:\Vamsi_IMP_Documents\test.csv')
thread_count=10
with ThreadPoolExecutor(max_workers=thread_count) as executor:
executor.map(export_data,mdx)

dataframe is misspelt?
data=tm1.cubes.cells.execute_mdx_datafrmae(expression)

Tq  jrobinsonAG , Yeah i spot that and corrected but still i am not getting a CSV with Data

have you checked the mdx statement against the cube directly, via PAfE or another interface?

yes i have checked MDX is working fine with one Department  as condition

what is the state of the data object just prior to the data.to_csv operation?

you may need to use a more robust string expression for your file name.
for instance escaping the ''
data.to_csv('C:\Vamsi_IMP_Documents\test.csv')

what is the state of the data object just prior to the data.to_csv operation?

it is data frame

you may need to use a more robust string expression for your file name.
for instance escaping the \
data.to_csv('C:\\Vamsi_IMP_Documents\\test.csv')

i have modified still the same issue

Just confirming, is the MDX statement that one that is being executed? I think you might be missing * on the Skills and Location lines

I would remove your file path and just put a file name like test.csv. It will put the file in your py directory. Chances are you either are not working in the directory you think you are, or you don't have proper permissions to write to the location.
Sent from my mobile phone
On Jun 22, 2021 10:48 PM, vamsi-cloud @.***> wrote:
you may need to use a more robust string expression for your file name.
for instance escaping the
data.to_csv('C:\Vamsi_IMP_Documents\test.csv')
i have modified still the same issue

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/cubewise-code/tm1py/issues/284#issuecomment-866547298, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEK7GZXR3KG62NJYHZP6BCLTUFYPPANCNFSM4O2W55GA.

@vamsi-cloud May be you need to revisit your use case here. If you just want to export the data, you can use unbound TI process to export the data into a csv file.
If it is just exporting a cube into a csv file, do you even need TM1py?
Also, you never mentioned if you received any error. If it is out of memory that you are getting, then you need to break down the MDX further.

@vamsi-cloud May be you need to revisit your use case here. If you just want to export the data, you can use unbound TI process to export the data into a csv file.
If it is just exporting a cube into a csv file, do you even need TM1py?
Also, you never mentioned if you received any error. If it is out of memory that you are getting, then you need to break down the MDX further.
@rkvinoth  actaully i can take CSV from tm1 to fetch the data but this mdx is not just to fetch data but after this we are performing operations on top of this data .and whole point of using MDX is to automate the Script  in the first step we are facing this issue so thought of passing it in threads to process

In a normal TI process you can use prolog to create the source view, in the data tab export the data into a csv file.
In the epilog call the python script which will load this file into memory and do the work for you.


Add information about HS256

Some people get confused about ssh keys (especially windows users) so I think it could be useful.

I think this needs a rebase.


squashfs-tools-ng: init at 0.7

Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Notify maintainers
cc @

Failure on macos 10.13: https://gist.github.com/risicle/0156756c1902a8d79a116856e6edf197
Related to similar sys/sysmacros.h issue in #68275?

Possibly telling https://gist.github.com/risicle/aa6fb3ccd0f962540c9be0b6cb12635c


[SPARK-38316][SQL][TESTS] Fix SQLViewSuite/TriggerAvailableNowSuite/Unwrap*Suite under ANSI mode

What changes were proposed in this pull request?

Fix the following test suites under ANSI mode:

TriggerAvailableNowSuite
GlobalTempViewTestSuite
LocalTempViewTestSuite
PersistedViewTestSuite
SimpleSQLViewSuite
UnwrapCastInComparisonEndToEndSuite
UnwrapCastInBinaryComparisonSuite

Why are the changes needed?

To set up a new GA test job with ANSI mode on
Does this PR introduce any user-facing change?

No
How was this patch tested?

Manually turn on ANSI mode and test .
Also it should pass GA tests.

@cloud-fan @HyukjinKwon @HeartSaVioR  Thanks for the reviews!
Merging to master


ParseNumbers works ugly

Hello, authors!
I have the form with inputs which has types "text" and "number".
When I use serializeJSON() method with argument "ParseNumbers:true", your plugin parse me all data which user input. (also data with type=text).
When I set ParseNumbers:false, your plugin doesn't parse anything..There is ability to depend of "type" of input? Thanks!

The plugin doesn't look at the input type. The types that the plugin handles are self-defined in the plugin.
See the :types section in the README.
All fields are parsed as :string by default. The parseNumbers: true option automatically detects and convert strings like "1", "33.33", "-44" to numbers like 1, 33.33, -44. None of this has anything to do with the input type. The input type is for HTML5 browsers.
It seems like you want to define your own types, do it with the :types sufix like this:
<input type="text"   name="username:string" value="Imperat"/>
<input type="number" name="issues:number"   value="1"/>

$('input').serializeJSON();

// returns =>
{
    "username": "Imperat",
    "issues": 1
}

Ok, now it's clear for me. Thanks for your reply!


mismatch on %sql charindex

reproduce and solve this reported issue

tip: the problem is most likely in the parsing logic. We can test it in test_command.py

Yes, it seems like the root of the problem is with the parsing logic (which happens only when trying to find the index of an empty char - ' ')
It happens here at parse_argstring(magic_execute, line).
i.e:
Given this line
line = "SELECT CHARINDEX(' ', 'AAA BBB CCC')"

It parses it to
['SELECT', "CHARINDEX('", "', '", 'AAA', 'BBB', "CCC')"]

which leads to this line (where the first empty char is before 'AAA')
SELECT CHARINDEX(' ', ' AAA BBB CCC')

Is it something that we can override? and if so, do we want to do it?
In the meantime, I tried this weird walkaround that so far worked and passed all the current tests.
If the given command is not a cell command and no args were given, so we transform the line into a cell and remove the line.
@edublancas @idomic
let me know what you think

Can we override it?
I think we can and we should.

In the meantime, Since the cell is working OK, I tried this weird walkaround that so far worked and passed all the current tests.

Is it possible there's no test cases covering this use case?

If the given command is not a cell command and no args were given, so we transform the line into a cell and remove the line.

@yafimvo I'm not so clear on this workaround, what's the result of the execution?

I think we can and we should.

I'm not sure which function to override. The parse_argstring comes from IPython
from IPython.core.magic_arguments import parse_argstring

@yafimvo I'm not so clear on this workaround, what's the result of the execution?
I think in general we should have the same behavior for %sql and %%sql, given the later is outputting the right index, we should mimick this behavior imo.

Everything remains the same for the user, %sql and %%sql. It just executes some line queries as cell queries.

I'm confused about the state of this. The original issue reported that the behavior of %sql and %%sql was different but based on @yafimvo's screenshot looks like they behave the same (?)

@edublancas the screenshot is after my walkaround.
The root of the problem is bad parsing when using %sql with this specific query (trying to find the empty char).

Ok, I understand the problem now. The parse_argstring from IPython.core.magic_arguments was designed to parse shell-like arguments. Example:
{program} {command} arg1 arg2 -a --something value

The problem is that users might pass an embedded SQL query:
{program} {command} arg1 arg2 -a --something value {some-sql-query-here}

And the query might contain spaces, leading to errors in parsing.
This isn't a simple fix since we're essentially using IPython's argparser for something it wasn't designed to do. I'd suggest keeping this in the backlog, since it doesn't seem to be a critical issue.

@edublancas Why don't we use this workaround? It aligns the behavior.

Because the workaround involves the user changing their SQL query.
From this:
line = "SELECT CHARINDEX(' ', 'AAA BBB CCC')"

to this:
line = "SELECT CHARINDEX(' ', ' AAA BBB CCC')"

and this is information that users don't know (they don't know there is a parser error and they should add this weird extra leading space).

@edublancas The user changes nothing. He executes the same query with %sql but behind the scene, we execute it as %%sql (instead of line we execute as cell).
i.e:
%%sql
SELECT CHARINDEX(' ', 'AAA BBB')

%sql SELECT CHARINDEX(' ', 'AAA BBB')

both these runs will return 4

I see. but the implementation might be a problem, right? how do we know when we execute %sql as %%sql?

Maybe 1 easy rule is if it contains a space.
Is there functionality that one has over the other?
I assumed it was just a styling matter.

I don't think it's a long term solution but more of a workaround so I see your point, maybe it's better to fix it ground up.

Space isn't enough. A %sql command can contain valid spaces for arguments.

yes, spaces would be enough here. I can push a PR with more tests so you'll see it.
It's not a long-term solution but passed all the current tests. The logic here was if it's %sql without arguments (which means just a query to execute) - run it as if it was a cell %%sql.
something like this
        if not cell:
            if not _any_arg_given:
                cell = line
                line = ""

Ok, let's see the tests and decide

Closing this issue given its an edge case and we need more user feedback on it.


Allow builds of pull requests from forks

Pull requests from forks fail due to security restrictions with secret variables (see https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions).
Integration tests and artifact signing rely on secret variables and thus must be disabled for builds on pull requests from forks.

Artifact signing is only used for builds on master or tags, thus can be disabled without impact. Disabling integration tests is more problematic, as it allows merging changes to master without ensuring all integration tests are green.

Builds from pull requests now only perform basic maven build, without executing integration tests that depend on secret variables.
In order to counter potential quality issues, pull requests from forks will be merged into a contributions branch first. After successful integration test execution on the contributions branch, the contributions branch will be merged (or rebased) into the master.


[autoversion] MEN-4366: Extend tool to support new Docker versions

As documented, when an AUTOVERSION tag requires the Docker version
rather than the default Git version, it shall append a /docker to the
action.
Updated the backend services versions accordingly.

Related to https://github.com/mendersoftware/integration/pull/895

I will open a new PR. This one can stay here in case we need this piece of code in the future.


multiversioning: fix aarch64 macos build

At first glance, you'd think this would have made the 0.15.5 release broken on macOS - but it turns out not!
0.15.5 pulled in the current release for 0.15.4 as a past release. That doesn't rely on offsets within a pack, so it worked.
0.15.3 was pulled from the pack. This would be affected, however the 0.15.3 embedded had one special quality: it was the universal binary itself for both platforms! This meant it was identical in size, offset and checksum, so using the x86_64 offsets and sizes and checksums for arm worked.

I would like to additionally assert the inactive header value checksums in print_information which is used by the CLI verification of the final built binaries. Since currently it's a blind spot.
Verifying the release on macos might be good too.

I would like to additionally assert the inactive header value checksums in print_information which is used by the CLI verification of the final built binaries - since currently it's a blind spot

It seems like it might be better to just run verification on both flavors of mac, so that you only verify the host arch, but we actually make sure to verify on all relevant hosts?


Delete unnecessary spaces

Delete unnecessary spaces: When copying from the web page, there is an unnecessary space after the code.
Fix indent: Indentation is misaligned.

@changeworld : Thanks for your contribution! The author, @yujhongmicrosoft, has been notified to review your proposed change.

@changeworld : Thanks for your contribution! The author, @yujhongmicrosoft, has been notified to review your proposed change.

@changeworld : Thanks for your contribution! The author, @yujhongmicrosoft, has been notified to review your proposed change.

@changeworld - Thank you for your submission. We are not able to accept formatting and spacing changes in the repo at this time.


0d3d703e - not sure... (spoiler)

I'll admit I was looking for easy ones (3x3)... Definitely hard on the "rule making" brain muscle.  Seemed to be loosely based on color scarcity and previous position.  I narrowed it down to brick, gray, yellow.  I had them in that order, and was hesitant because gray had already appeared in that position...  I clicked "check" and it worked.  So now I wonder, was I mistaken and lucky? or was this something different...  Maybe all my rules were just wrong and I was super lucky.  So, not knowing the actual rules, I don't know if this was an error or not, but it seemed somehow wrong.  I tried swapping brick and gray, but that failed.  I know it's just one test.  I'd pay money to know if an LLM got this, ever!

Basically I chose [brick red], [gray], [yellow], even though I felt that the rules didn't quite support that choice.  It was "correct", but it "should not have been correct", IMO.

ARC-Interactive

it's color replacements:

green -> yellow
blue -> grey
red -> purple
cyan -> brown
grey -> blue
purple -> red
brown -> cyan
yellow -> green

IMHO this is a trivial task. I think this issue can be closed.


SHA Acc premature release in ROM

Closing this PR as it is no longer applicable due to https://github.com/chipsalliance/caliptra-sw/pull/862


Getting profile or map should not require scope

Description

This PR fixes getProfile, getProfileAST, getProfileSource, getMap, getMapAST, getMapSource functions. Parameters scope and version are no longer required to get profile or map resources.
BREAKING CHANGE Parameters of getProfile* and getMap* functions has been changed.
Motivation and Context

Fixes: https://github.com/superfaceai/service-client/issues/67
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTION_GUIDE document.
[x] I haven't repeated the code. (DRY)
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

@kysely I checked Air and it does work well with profiles without scope.

why keep the profile id components separate? Why not simply expect a valid profileId

I think it is CLI requirement. @Jakub-Vacek can you confirm that CLI works with profile id components and not full profileId.

why keep the version required? A request for the latest without specifying a version seems valid to me.

Yes that makes sense to me too. @Jakub-Vacek what do you think?

Currently, CLI http functions (profile and map related) calls directly ServiceClient.fetch where it passes profileId as a string. On the other places we use profileId as ProfileId and optional (which is imo correct - user does not have to pass version expecting that he gets the latest profile/map) version.
CLI will be able to work with both approaches with very little effort. I think the question is more about what is the responsibility of service client. When we decide to pass profile components it will be responsible for construction of the URL when we will pass valid profile ID this responsibility is on CLI/AIR.

The other question is whether ServiceClient should expect its users to understand the profile/map IDs components. But since this is aimed for use in CLI/front end, I think it's reasonable.
In that case I'd suggest keeping the components separate but still make the version optional.

@Jakub-Vacek @kysely I made version optional parameter. 🙏 review once again.

Oh I missed that version is required in case of map. Thank you!

putting all of the document ID components into an object seems like an easier API for me

I wanted to pass optional parameters as options, but as all of the parameters are actually document ID components your suggestion makes sense.


Add CC machine options

Closes #9010.

This change is

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/febca780a16733456d691b54c5486607fbfdc2fb/

LGTM @lnhsingh !

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/4c552edb763a6e2ffcc3b7c52c13782a51812b9a/


Update repo pages

Resolved / Related Issues
None
Details of Changes

Added some pages for repository.

Validation
How did you test these changes?

[x] Built and ran the app
[ ] Tested the changes for accessibility

Screenshots (optional)
Add screenshots here.

Do not marge. still in work.

Nice!!!

Marge now!!!
LGTM.


check that caps are valid

Finally got around to looking at this too.
Rather than build the image and have something weird happen, let's check that the capabilities specified are actually valid capabilities.

Sorry I missed this when travelling. It needs a rebase since I split out things into a library

Done, thanks.


feat: complete order information about products and delivery point

Pull Request

Description

Type of Change

[x] ✨ New feature (non-breaking change which adds functionality)
[ ] 🛠 Bug fix (non-breaking change which fixes an issue)
[ ] ❌ Breaking change (fix or feature that would cause existing functionality
to change)
[ ] 🧹 Code refactor
[ ] ✅ Build configuration change
[ ] 📝 Documentation
[ ] 🗑 Chore

Task linked: CU-8689qdx16 Complete order information about products and delivery point


Error loading python in WSL

Environment data

VS Code version: 1.60.1
OS and version: Windows Feature Experience Pack 1000.22458.1000.0
Python version (& distribution if applicable, e.g. Anaconda): python3.8
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv
Relevant/affected Python-related VS Code extensions and their versions: 2021.9.1230869389
Value of the python.languageServer setting: python3.8

Expected behaviour
Load python extension in WSL remote
Actual behaviour
Python is  ignoring  my configuration and start to load conda and pyenv
Steps to reproduce:
Create one project in WSL and try load python by remote WSL
Logs
User belongs to experiment group 'pythonaacf'
User belongs to experiment group 'pythonJediLSP'
User belongs to experiment group 'pythonDiscoveryModuleWithoutWatcher'
User belongs to experiment group 'pythonTensorboardExperiment'
User belongs to experiment group 'PythonPyTorchProfiler'
User belongs to experiment group 'pythonDeprecatePythonPath'
User belongs to experiment group 'pythonSortEnvs'
User belongs to experiment group 'pythonRunFailedTestsButtonDisplayed'
User belongs to experiment group 'pythonRefreshTestsButtonDisplayed'
User belongs to experiment group 'pythonRememberDebugConfigcf'

conda --version
pyenv root
python3.7 -c "import sys;print(sys.executable)"
python3.6 -c "import sys;print(sys.executable)"
python3 -c "import sys;print(sys.executable)"
python2 -c "import sys;print(sys.executable)"
python -c "import sys;print(sys.executable)"
~/Worker/NIC/nic-production/.venv/bin/python -c "import sys;print(sys.executable)"
conda info --json

The workaround mentioned in https://github.com/microsoft/vscode-python/issues/14977#issuecomment-919588921 should fix the issue for you, we plan to make a point release with the fix today. Do note you need to use Remote settings instead of User settings if you're using WSL.
Closing as dup of https://github.com/microsoft/vscode-python/issues/14977


[Bloc] Element layout alignments are not stable

We get different position after each requestLayout:

works now


Nginx symlink configuration

Hello,
Sorry for this new question, maybe not related to this app.
I've managed to setup the deploy script, but but pointing my vhost in nginx to current/public, it doesn't work.
Though, when I point to the current latest release in the releases folder, then it does work.
I've tried setting disable_symlinks on; in nginx.conf and the vhost, but doesn't change anything.
I'm sure it's something rather stupid..
Thanks!

 I must have missed something, just had to put that disable_symlinks offl in the vhost instead of nginx.conf
Sorry again for the trouble,


[DO NOT MERGE][BEAM-5627][BEAM-6569] run gcsio tests

Run gcsio tests to find cause of issues flaky tests
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Run Python PreCommit

Run Python PreCommit

LGTM, thank you!


Metal ion binding dataset

Hi there,
Nice work!
I have a question about the metal ion binding dataset used in your paper.
Could you let me know where do you get the original dataset?
Thanks!

Hi, empyriumz:
Metal ion binding dataset collected from PDB(https://www.rcsb.org/). If the protein has any Metal ion binding site, we set its label as 1.

Thanks for your reply!
To clarify, I tries to search on PDB for metal ion binding:

which results in 87,669 entries.
Do you also perform similar queries and compile the dataset?

By original dataset, do you mean all the PDB files? That would be too large I guess, so could you share the script used for search and annotate the PDB entries?
Thanks!

I am so sorry that the classmates who wrote the crawler are not on the author list and are unwilling to give it to us. They now have a job and will also release the relevant dataset. I can notify you after their paper is released.
But I can give you a simple code that can check whether each page contains keywords. It may help you.
    url = 'https://www.rcsb.org/annotations/2XEV'     req = urllib.request.Request(url=url)     content = urllib.request.urlopen(req).read()      content=content.decode('utf-8')      soup = BeautifulSoup(content,"html.parser")     tag = soup.find_all(text='metal ion binding')

Hi, I try to use your metal alphafold code to predict other protein features, but I find that your code use a pkl data as the input, so I want to know how you generate the pkl files.Thanks!

Hi Violet969:
This pkl including MSA and template information.
Related code you can see https://github.com/deepmind/alphafold/blob/main/run_alphafold.py line 172-174. When data_pipeline.process input a fasta and it will return MSA, template and pkl.
feature_dict = data_pipeline.process(
input_fasta_path=fasta_path,
msa_output_dir=msa_output_dir)
Pkl detail information you can see Alphafold paper's supplementary information pages 8-9.
I have already released the MSA on https://drive.google.com/drive/folders/1iShEW8NcMIlWqxTRgsEaI_t5ahoHsixt?usp=share_link
But the code to generate pkl maybe you need to modify some on run_alphafold.py. I can upload this part of the preprocessing code later.

I see, thanks for your sample code! I'll try to see if the results match with my aforementioned one.

Thanks for your answer. I also have a question, I saw that you use Evofomer and ESM to predict protein SS. But I don't see these code, will you share that?

Thanks for your answer. I also have a question, I saw that you use Evofomer and ESM to predict protein SS. But I don't see these code, will you share that?

Sure, I will upload this part of the code later.

Hi Violet969,
Secondary structure related codes and the code that can generate pkl from a3m have been uploaded into the Structure folder and Data folder, if you have any questions you can contact me.

Thanks for your answer. I also have a question, I saw that you use Evofomer and ESM to predict protein SS. But I don't see these code, will you share that?

Hi Violet969,
Secondary structure related codes and the code that can generate pkl from a3m have been uploaded into the Structure folder and Data folder, if you have any questions you can contact me.

I see, thanks for the answer. I used merge_msa.py but it didn't work, can you show me a case how to use it?

I see, thanks for the answer. I used merge_msa.py but it didn't work, can you show me a case how to use it?

Hi,
I have added an example, you can have a look at the latest code.

I see, thanks for the answer. I used merge_msa.py but it didn't work, can you show me a case how to use it?

Hi, I have added an example, you can have a look at the latest code.

Thanks for your answer. I also want an example for run metal/alphafold/train.py. Can you share that?

I see, thanks for the answer. I used merge_msa.py but it didn't work, can you show me a case how to use it?

Hi, I have added an example, you can have a look at the latest code.

Thanks for your answer. I also want an example for run metal/alphafold/train.py. Can you share that?

I see, thanks for the answer. I used merge_msa.py but it didn't work, can you show me a case how to use it?

Hi, I have added an example, you can have a look at the latest code.

Thanks for your answer. I also want an example for run metal/alphafold/train.py. Can you share that?

Now you should be able to run train.py directly with a few simple modifications. Please make sure you have configured the Alphafold runtime environment.
In addition, it seems that the current Alphafold parameter format is different from before. You can try to find the previous public parameter file.

Thanks for your reply. I try to run 'train.py' on my server. But there always have an error like this.

2023-01-01 07:07:23.507834: E external/org_tensorflow/tensorflow/compiler/xla/pjrt/pjrt_stream_executor_client.cc:2130] Execution of replica 0 failed: INTERNAL: Failed to allocate 50331648 bytes for new constant
Traceback (most recent call last):
File "train.py", line 264, in
app.run(main)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "train.py", line 216, in main
state = jax.pmap(updater.init)(rng_pmap, data)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/_src/traceback_util.py", line 162, in reraise_with_filtered_traceback
return fun(*args, **kwargs)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/src/api.py", line 2158, in cache_miss
out_tree, out_flat = f_pmapped(*args, **kwargs)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/_src/api.py", line 2034, in pmap_f
out = pxla.xla_pmap(
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 2022, in bind
return map_bind(self, fun, *args, **params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 2054, in map_bind
outs = primitive.process(top_trace, fun, tracers, params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 2025, in process
return trace.process_map(self, fun, tracers, params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 687, in process_call
return primitive.impl(f, *tracers, **params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/interpreters/pxla.py", line 841, in xla_pmap_impl
return compiled_fun(*args)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/_src/profiler.py", line 294, in wrapper
return func(*args, **kwargs)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/interpreters/pxla.py", line 1656, in call
out_bufs = self.xla_executable.execute_sharded_on_local_devices(input_bufs)
jax._src.traceback_util.UnfilteredStackTrace: jaxlib.xla_extension.XlaRuntimeError: INTERNAL: Failed to allocate 50331648 bytes for new constant: while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well).
The stack trace below excludes JAX-internal frames.
The preceding is the original exception that occurred, unmodified.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "train.py", line 264, in
app.run(main)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "train.py", line 216, in main
state = jax.pmap(updater.init)(rng_pmap, data)
jaxlib.xla_extension.XlaRuntimeError: INTERNAL: Failed to allocate 50331648 bytes for new constant: while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well).

I have 8 nodes of 12G GPU, and 125G mem. Can you tell me how to solve it?

Thanks for your reply. I try to run 'train.py' on my server. But there always have an error like this.

2023-01-01 07:07:23.507834: E external/org_tensorflow/tensorflow/compiler/xla/pjrt/pjrt_stream_executor_client.cc:2130] Execution of replica 0 failed: INTERNAL: Failed to allocate 50331648 bytes for new constant
Traceback (most recent call last):
File "train.py", line 264, in
app.run(main)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "train.py", line 216, in main
state = jax.pmap(updater.init)(rng_pmap, data)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/_src/traceback_util.py", line 162, in reraise_with_filtered_traceback
return fun(*args, **kwargs)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/src/api.py", line 2158, in cache_miss
out_tree, out_flat = f_pmapped(*args, **kwargs)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/_src/api.py", line 2034, in pmap_f
out = pxla.xla_pmap(
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 2022, in bind
return map_bind(self, fun, *args, **params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 2054, in map_bind
outs = primitive.process(top_trace, fun, tracers, params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 2025, in process
return trace.process_map(self, fun, tracers, params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 687, in process_call
return primitive.impl(f, *tracers, **params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/interpreters/pxla.py", line 841, in xla_pmap_impl
return compiled_fun(*args)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/_src/profiler.py", line 294, in wrapper
return func(*args, **kwargs)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/interpreters/pxla.py", line 1656, in call
out_bufs = self.xla_executable.execute_sharded_on_local_devices(input_bufs)
jax._src.traceback_util.UnfilteredStackTrace: jaxlib.xla_extension.XlaRuntimeError: INTERNAL: Failed to allocate 50331648 bytes for new constant: while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well).
The stack trace below excludes JAX-internal frames.
The preceding is the original exception that occurred, unmodified.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "train.py", line 264, in
app.run(main)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "train.py", line 216, in main
state = jax.pmap(updater.init)(rng_pmap, data)
jaxlib.xla_extension.XlaRuntimeError: INTERNAL: Failed to allocate 50331648 bytes for new constant: while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well).

I have 8 nodes of 12G GPU, and 125G mem. Can you tell me how to solve it?

I tested this code on A40(48GB) server and it works.
You can try to set " os.environ['XLA_PYTHON_CLIENT_MEM_FRACTION'] = '2' " or lower to reduce memory usage.

Thanks for your reply. I try to run 'train.py' on my server. But there always have an error like this.

2023-01-01 07:07:23.507834: E external/org_tensorflow/tensorflow/compiler/xla/pjrt/pjrt_stream_executor_client.cc:2130] Execution of replica 0 failed: INTERNAL: Failed to allocate 50331648 bytes for new constant
Traceback (most recent call last):
File "train.py", line 264, in
app.run(main)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "train.py", line 216, in main
state = jax.pmap(updater.init)(rng_pmap, data)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/_src/traceback_util.py", line 162, in reraise_with_filtered_traceback
return fun(*args, **kwargs)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/src/api.py", line 2158, in cache_miss
out_tree, out_flat = f_pmapped(*args, **kwargs)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/_src/api.py", line 2034, in pmap_f
out = pxla.xla_pmap(
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 2022, in bind
return map_bind(self, fun, *args, **params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 2054, in map_bind
outs = primitive.process(top_trace, fun, tracers, params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 2025, in process
return trace.process_map(self, fun, tracers, params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 687, in process_call
return primitive.impl(f, *tracers, **params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/interpreters/pxla.py", line 841, in xla_pmap_impl
return compiled_fun(*args)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/_src/profiler.py", line 294, in wrapper
return func(*args, **kwargs)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/interpreters/pxla.py", line 1656, in call
out_bufs = self.xla_executable.execute_sharded_on_local_devices(input_bufs)
jax._src.traceback_util.UnfilteredStackTrace: jaxlib.xla_extension.XlaRuntimeError: INTERNAL: Failed to allocate 50331648 bytes for new constant: while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well).
The stack trace below excludes JAX-internal frames.
The preceding is the original exception that occurred, unmodified.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "train.py", line 264, in
app.run(main)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "train.py", line 216, in main
state = jax.pmap(updater.init)(rng_pmap, data)
jaxlib.xla_extension.XlaRuntimeError: INTERNAL: Failed to allocate 50331648 bytes for new constant: while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well).

I have 8 nodes of 12G GPU, and 125G mem. Can you tell me how to solve it?

I tested this code on A40(48GB) server and it works. You can try to set " os.environ['XLA_PYTHON_CLIENT_MEM_FRACTION'] = '2' " or lower to reduce memory usage.

Thanks for your so fast reply, that 'os.environ['XLA_PYTHON_CLIENT_MEM_FRACTION'] = '2'' works.
But i met another error like these
Traceback (most recent call last):
  File "train.py", line 269, in <module>
    app.run(main)
  File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "train.py", line 233, in main
    state, metrics = updater.update(state, data)
  File "train.py", line 176, in update
    if step % self._checkpoint_every_n == 0:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Can you tell me how to solve it?

Thanks for your reply. I try to run 'train.py' on my server. But there always have an error like this.

2023-01-01 07:07:23.507834: E external/org_tensorflow/tensorflow/compiler/xla/pjrt/pjrt_stream_executor_client.cc:2130] Execution of replica 0 failed: INTERNAL: Failed to allocate 50331648 bytes for new constant
Traceback (most recent call last):
File "train.py", line 264, in
app.run(main)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "train.py", line 216, in main
state = jax.pmap(updater.init)(rng_pmap, data)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/_src/traceback_util.py", line 162, in reraise_with_filtered_traceback
return fun(*args, **kwargs)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/src/api.py", line 2158, in cache_miss
out_tree, out_flat = f_pmapped(*args, **kwargs)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/_src/api.py", line 2034, in pmap_f
out = pxla.xla_pmap(
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 2022, in bind
return map_bind(self, fun, *args, **params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 2054, in map_bind
outs = primitive.process(top_trace, fun, tracers, params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 2025, in process
return trace.process_map(self, fun, tracers, params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/core.py", line 687, in process_call
return primitive.impl(f, *tracers, **params)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/interpreters/pxla.py", line 841, in xla_pmap_impl
return compiled_fun(*args)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/_src/profiler.py", line 294, in wrapper
return func(*args, **kwargs)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/jax/interpreters/pxla.py", line 1656, in call
out_bufs = self.xla_executable.execute_sharded_on_local_devices(input_bufs)
jax._src.traceback_util.UnfilteredStackTrace: jaxlib.xla_extension.XlaRuntimeError: INTERNAL: Failed to allocate 50331648 bytes for new constant: while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well).
The stack trace below excludes JAX-internal frames.
The preceding is the original exception that occurred, unmodified.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "train.py", line 264, in
app.run(main)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "train.py", line 216, in main
state = jax.pmap(updater.init)(rng_pmap, data)
jaxlib.xla_extension.XlaRuntimeError: INTERNAL: Failed to allocate 50331648 bytes for new constant: while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well).

I have 8 nodes of 12G GPU, and 125G mem. Can you tell me how to solve it?

I tested this code on A40(48GB) server and it works. You can try to set " os.environ['XLA_PYTHON_CLIENT_MEM_FRACTION'] = '2' " or lower to reduce memory usage.

Thanks for your so fast reply, that 'os.environ['XLA_PYTHON_CLIENT_MEM_FRACTION'] = '2'' works. But i met another error like these
Traceback (most recent call last):
  File "train.py", line 269, in <module>
    app.run(main)
  File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/anaconda3/envs/alphafold_2/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "train.py", line 233, in main
    state, metrics = updater.update(state, data)
  File "train.py", line 176, in update
    if step % self._checkpoint_every_n == 0:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Can you tell me how to solve it?

Delete the './tmp' folder.


BUILDKITE_PULL_REQUEST is not correctly detected

I have dynamic pipeline to run some checks only if it's PR.
And the checks are mandatory before merging the PR on github.
Agent Version | 2.3.2
And sometimes the BUILDKITE_PULL_REQUEST value is false evening merging changes into the feature branch.

This sounds like a pull requests cache issue. @sj26 / @toolmantim?

Oh dear, sorry that doesn't sound good :-(
I agree this doesn't sounds like an agent issue. @exiadb, could you email us the specifics of this case to<EMAIL_ADDRESS>including links to the builds with which you're having trouble? Meanwhile I'll close this issue here.

We're experiencing this bug as well. It seems to fail about half of the time.

It’s probable that we are receiving the branch / commit notification from GitHub before we get the pr:open notification, which means the build is already created. Subsequent commits to the branch should always show up with a correct BUILDKITE_PULL_ENV. Does that sound right?
Drop us an email with an example build to<EMAIL_ADDRESS>and we will investigate!

On 17 Nov 2017, at 8:20 am, Brad Wilson<EMAIL_ADDRESS>wrote:
We're experiencing this bug as well. It seems to fail about half of the time.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Yes, after reviewing, that's exactly what we've been seeing. It's always the first commit. Do you have any suggestions for how to mitigate?

It's a very tricky problem to solve, as quite often the two events are separated by quite some time as developers generally push a branch first and then create a pull request from that branch.
We have been discussing ways to mitigate this from a UX perspective, for instance updating the page dynamically when the PR comes in, but trying to get it into the environment vars is a hard problem to solve in the sense that the variables are out of our control.
About the only practical way to ensure you only get builds for pull requests is to only trigger builds on pull requests or on master. This is what we do for some of our repositories and it works really well provided your team has a pull-request centric workflow. This is what our config looks like:

That actually works perfectly for us. Thank you!


Calculate mesh normals - the STL file might not include them

Hey,
This fixes an issue when some files are rendered "all black" even when the color is set. This happens because of missing normals data that the shader relies on.

Great @asafge thanks!