Fix the dataset import

Fix https://github.com/torchmd/torchmd-net/pull/83#discussion_r873772964

@PhilippThoelke could you review?


Reduce white space on left, and possibly base font size

While integrating recent visual changes to guides repo some issues were noticed:

Lots of horizontal space lost
Content height grows significantly
Lots of white space

See: https://percy.io/e5d9f90c/guides/builds/15507671/changed/876102287?browser=edge&browser_ids=18%2C19%2C20%2C21&subcategories=unreviewed%2Cchanges_requested&viewLayout=side-by-side&viewMode=new&width=375&widths=375

Suggestion: Align bullet points with paragraphs in procedure to reduce white space used. https://percy.io/e5d9f90c/guides/builds/15507671/changed/876102270?browser=edge&browser_ids=18%2C19%2C20%2C21&subcategories=unreviewed%2Cchanges_requested&viewLayout=side-by-side&viewMode=new&width=375&widths=375

Fixed with design refresh


Add "impact_statement" to specification

The current Minimum Requirements for VEX document requires an "impact_statement" when justification is "not_affected" or similar. Practically, a security auditor may want to look at the reason the author applied a justification and either add their own vex statement or revert the statement.

Here's the relevant excerpt from that doc (section <IP_ADDRESS>.1):

For [status] “not_affected”, if [justification] is not provided, then a VEX statement MUST provide an [impact_statement] that further explains how or why the listed [product_id]s are “not_affected” by [vul_id].

I believe the OpenVEX spec complies with the Minimum Requirements already. The Minimum Requirements require an impact_statement only when BOTH the status is "not_affected" AND when a justification is not provided.
Here's the relevant excerpt from the OpenVEX spec:

For statements conveying a not_affected status, a VEX statement MUST include either a status justification or an impact_statement informing why the product is not affected by the vulnerability. An impact statement is a free form text containing a description of why the vulnerability cannot be exploited. This field is not intended to be machine readable so its use is highly discouraged for automated systems.


Issue with Jz calculation

Hi,
I am running into the following issue while trying to estimate Jz based on the notebook tutorial
Here is my code
import zoomies
from astroquery.gaia import Gaia
from astropy.coordinates import SkyCoord
c=SkyCoord('13:44:25.4773 −40:20:15.5222', unit=(u.hourangle, u.deg))
r=Gaia.query_object(c, radius=5*u.arcsec)
r_actions = zoomies.calc_jz(r, method='agama')

I get the following error
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[46], line 2
      1 c=SkyCoord('13:44:25.4773 −40:20:15.5222', unit=(u.hourangle, u.deg))
----> 2 r_actions = zoomies.calc_jz(r, method='agama')
      3 r_actions

File ~/anaconda3/lib/python3.11/site-packages/zoomies/quick_jz.py:67, in calc_jz(gaia_table, method, mwmodel, write, fname)
     64 import agama
     66 agama.setUnits(mass=u.Msun, length=u.kpc, time=u.Myr)
---> 67 agama_pot = mw.as_interop("agama")
     68 af = agama.ActionFinder(agama_pot)
     69 Jr, Jz, Jphi = af(w.w(galactic).T).T * 1000 # agama units are different from galpy

AttributeError: 'MilkyWayPotential2022' object has no attribute 'as_interop'

Also, when I try using galpy, there is an error
r_actions = zoomies.calc_jz(r, method='galpy')

File ~/anaconda3/lib/python3.11/site-packages/zoomies/quick_jz.py:55, in calc_jz(gaia_table, method, mwmodel, write, fname)
     49 w = gd.PhaseSpacePosition(galcen.data)
     51 if method=='galpy':
     52
     53     # print('Calculating actions with galpy...')
---> 55     aaf = galpy_find_actions_staeckel(mw, w)
     56     Jz = aaf['actions'][:, 2]
     57     Jphi = aaf['actions'][:, 1]

File ~/anaconda3/lib/python3.11/site-packages/gala/dynamics/actionangle/tests/staeckel_helpers.py:84, in galpy_find_actions_staeckel(potential, w, mean, delta, ro, vo)
     81 o = w_.to_galpy_orbit(ro, vo)
     82 aAS = actionAngleStaeckel(pot=galpy_potential, delta=delta_)
---> 84 aaf = aAS.actionsFreqsAngles(o)
     85 aaf = {
     86     "actions": np.array(aaf[:3]).T * ro * vo,
     87     "freqs": np.array(aaf[3:6]).T * vo / ro,
     88     "angles": coord.Angle(np.array(aaf[6:]).T * u.rad),
     89 }
     90 if mean:

File ~/anaconda3/lib/python3.11/site-packages/galpy/util/conversion.py:1220, in actionAngle_physical_input.<locals>.wrapper(*args, **kwargs)
   1217 @wraps(method)
   1218 def wrapper(*args, **kwargs):
   1219     if len(args) < 3:  # orbit input
-> 1220         return method(*args, **kwargs)
   1221     ro = kwargs.get("ro", None)
   1222     if ro is None and hasattr(args[0], "_ro"):

File ~/anaconda3/lib/python3.11/site-packages/galpy/util/conversion.py:1206, in physical_conversion_actionAngle.<locals>.wrapper.<locals>.wrapped(*args, **kwargs)
   1204     return newOut
   1205 else:
-> 1206     return method(*args, **kwargs)

File ~/anaconda3/lib/python3.11/site-packages/galpy/actionAngle/actionAngle.py:269, in actionAngle.actionsFreqsAngles(self, *args, **kwargs)
    245 """
    246 Evaluate the actions, frequencies, and angles (jr,lz,jz,Omegar,Omegaphi,Omegaz,angler,anglephi,anglez)
    247
   (...)
    266 - 2014-01-03 - Written for top level - Bovy (IAS)
    267 """
    268 try:
--> 269     return self._actionsFreqsAngles(*args, **kwargs)
    270 except AttributeError:  # pragma: no cover
    271     raise NotImplementedError(
    272         "'actionsFreqsAngles' method not implemented for this actionAngle module"
    273     )

File ~/anaconda3/lib/python3.11/site-packages/galpy/actionAngle/actionAngleStaeckel.py:386, in actionAngleStaeckel._actionsFreqsAngles(self, *args, **kwargs)
    384     R, vR, vT, z, vz, phi = args
    385 else:
--> 386     self._parse_eval_args(*args)
    387     R = self._eval_R
    388     vR = self._eval_vR

File ~/anaconda3/lib/python3.11/site-packages/galpy/actionAngle/actionAngle.py:155, in actionAngle._parse_eval_args(self, *args, **kwargs)
    153     orb = args[0]
    154 if len(orb.shape) > 1:
--> 155     raise RuntimeError(
    156         "Evaluating actionAngle methods with Orbit instances with multi-dimensional shapes is not supported"
    157     )
    158 self._eval_R = orb.R(use_physical=False)
    159 self._eval_vR = orb.vR(use_physical=False)

RuntimeError: Evaluating actionAngle methods with Orbit instances with multi-dimensional shapes is not supported

Hello,
Can you tell me what version of gala you're running -- and if it's not the most recent version, could you try updating the gala package and run the code again?
I believe the gala.potential.as_interop() function was added to the package recently, so if you're running version < 1.8 it might not include that function. I'll update the requirements for that -- thank you for pointing this out! Let me know if updating gala works.
As for galpy -- I have been having the same issue you've run into, and I should probably remove support for calculating action with galpy until I get that figured out. Using agama should still work though.
Sheila


Face liveness detection

While Trying to access with mobile photos , App recognition was granted. How do i make changes to not allow the mobile photos for accessing the application.

While Trying to access with mobile photos , App recognition was granted. How do i make changes to not allow the mobile photos for accessing the application.

Any update on this?

I don't really get the question, sorry. Can you explain a bit more?


LC0044 - False Positive on FlowFields

I'm unsure if the rule should raised in a scenario with FlowFields.
table 50100 MyTableA
{
    fields
    {
        field(1; MyField; Integer) { }
        field(2; "MyFlowField A"; Integer)
        {
            FieldClass = FlowField;
            CalcFormula = count(Customer);
        }
    }
}

table 50101 MyTableB
{
    fields
    {
        field(1; MyField; Integer) { }
        field(2; "MyFlowField B"; Integer)
        {
            FieldClass = FlowField;
            CalcFormula = count(Customer);
        }
    }
}

procedure MyProcedure()
var
    MyTableA: Record MyTableA;
    MyTableB: Record MyTableB;
begin
    MyTableA.TransferFields(MyTableB);
end;

Can we just skip all fields where FieldClass is set to FlowFilter or FlowField? 🤔

I don't think that FlowField to FlowField should raise this rule. They will be skipped anyway, and does not make any sense in TransferFields().
But there is also another interesting scenario, from FlowField to normal field.
table 50100 MyTableA
{
    fields
    {
        field(1; MyField; Integer) { }
        field(2; "MyField A"; Integer)
        {
        }
    }
}

table 50101 MyTableB
{
    fields
    {
        field(1; MyField; Integer) { }
        field(2; "MyFlowField B"; Integer)
        {
            FieldClass = FlowField;
            CalcFormula = count(Customer);
        }
    }
}

procedure MyProcedure()
var
    MyTableA: Record MyTableA;
    MyTableB: Record MyTableB;
begin
    MyTableB.CalcFields("MyFlowField B");
    MyTableA.TransferFields(MyTableB);
end;

In above scenario, "MyField A" would not get any value - even if "MyFlowField B" has a value. Should the rule be raised here?

FlowFields and FlowFilters are excluded in the (pre)release version of v0.30.11 of the LinterCop, whichs resolved this issue.


Do not detect Mac OS 9 new line on other system

It fix the issue gradle/gradle#1646 where Findbugs use Carriage Returns
to rewrite the current line.
Contributor Checklist

[x] Review Contribution Guidelines
[x] Sign Gradle CLA
[x] Link to Design Spec for changes that affect more than 1 public API (that is, not in an internal package) or updates to > 20 files
[x] Provide integration tests (under <subproject>/src/integTest) to verify changes from a user perspective
[x] Provide unit tests (under <subproject>/src/test) to verify logic
[x] Update User Guide, DSL Reference, and Javadoc for public-facing changes
[ ] Ensure that tests pass locally: ./gradlew quickCheck <impacted-subproject>:check

Gradle Core Team Checklist

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[x] Verify documentation including proper use of @since and @Incubating annotations for all public APIs
[x] Recognize contributor in release notes

@lacasseio quick note that we should mark PRs from our team as from:member to match GitHub's description

We count PRs from:contributor as external to know that our community is growing.

My bad wrongly tagging this PR. I always get confused when using those two from tag. It's really just the way I think about it so it was bound to happen at some point. I will be more careful in the future. I push a change and kick-started another CI run. You can review the code, I will make sure the CI pass this time.


Convert hard-coded link to Sphinx relative link

The Gulp doc contains a hard-coded link to the "Working with Multiple Environments" doc in the Fundamentals section. This PR converts the link to a relative Sphinx-style link, making it a bit easier to maintain.

Thanks! Merged in 9fadf62fe1c45449fe10e6413a834ffd46dae203


How change the Cp and IVs to my pokemons I need help

I put in the terminal this:
video: https://youtu.be/cii9XTO-AcA
PokemonGoMITM = require './lib/pokemon-go-mitm'
server = new PokemonGoMITM port: 8081

And before I don´t now what put! :(

I would like to improve my Pokemons just that and more XP
Example: I have a Dragonite with 2300 but I don´t know what put in the terminal to increase 3500 and IVs 100% :/

Someone who could put the code exactly increase my Dragonite  only that

please :c

A MITM proxy only changes what is displayed by your app. It doesn't change the actual values stored on the server, so changing IVs would be meaningless.

But there is a file to change pokemon and there you can edit it to your liking
https://github.com/rastapasta/pokemon-go-mitm/blob/master/example.replacePokemons.coffee


Aseprite command path invalid

command path is right then also it's showing command not found. my aesprite version is v1.2.39. is the addon not compatible with this version of aseprite.

Your two screenshots show different paths. The bottom one has an extra Program Files (x86) on it. That's likely the missing part.

Thank you very much it's working now.☺ I was really confused. Thanks a lot.


🧪 TESTS: Pin ipython<8

ipython v8 is only available for Python 3.8+, and it changes exception text, which fails some regression tests

Huh that minimum python version surprises me - I'd have assumed that ipython was following the NumPy/PyData NEP for version pinning: https://numpy.org/neps/nep-0029-deprecation_policy.html

https://github.com/ipython/ipython/blob/46a51ed69cdf41b4333943d9ceeb945c4ede5668/setup.cfg#L29

ya I saw that - I am just surprised that they decided to do it. I couldn't find an issue where they were discussing the rationale

Oh indeed yeh it is an interesting one

Ah maybe I got my dates wrong, the commit message where he added this references that NEP as well:
https://github.com/ipython/ipython/commit/5b9aa9c88325012439384b5a704e2b3d183c3366#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7

why does https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table decide to not use https://endoflife.date/python? seems like theres a strange offset

good question - I am asking around, I wasn't involved in those numpy conversations 🤷


Helper text for initial allocations

delma.betancourt left a comment:
Would be great to have more context and info on each of these screens - as a user I want to know whats happening and why, more context on each of these sections

Browser metadata
Path:      /create
Browser:   Chrome <IP_ADDRESS> on Mac OS 10.15.7
Viewport:  1307 x 958 @2x
Language:  en-US
Cookies:   Enabled

Open in BrowserStack

Open Deploy Preview · Mark as Resolved
Originally posted by @netlify[bot] in https://github.com/decentdao/fractal-interface/issues/1578#issuecomment-2117947161

I can't think of what other context we could put on this page. You have no right to be creating a DAO if you don't know what Name/Symbol/TotalSupply of an ERC20 token are, or if you don't understand the concept of minting that token to different addresses.
Closing.


Minificpp 927 Nanofi tailfile delimited processor

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file?
[ ] If applicable, have you updated the NOTICE file?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Rebased MINIFICPP-926 on this branch. So you will see changes from that branch as well in this PR. However, the only changes are in nanofi/src/api/ecu.c file and nanofi/ecu/tailfile_delimited.c

I tried the examples manually, but could not get it to work (it did not transmit flow files to my NiFi instance). After a little bit of debugging what I can see is that is it trying to use the wrapped MiNiFi implementation for sending flow files to the RPG, but that is missing a content repository and many other things.
I think it should use the same native implementation that is used in the transmit_payload example. @arpadboda might be able to help with it, since he wrote that.
However, it is a -1 from me until it actually works and I can try it.
Please also see my inline comments, especially about resolved issues which I think are not resolved.

@bakaid .. please review this again. I have fixed the transmission of flow files to nifi using the transmit_payload as an example.

It's getting better, I appreciate that it can now transmit flow files to NiFi.
I have still have some concerns to address:
-The mixed mode seems to introduce a lot of complexity but not sure it worth the effort, do you really think we need it? @phrocker what do you think?
-There is no logging even though Nanofi supports it. Could you take a look at site2site and logging part of nanofi for some inspiration?
-Given the number of allocations I'm a bit afraid of introducing leaks, could you try running valgrind or any other dynamic analytics tool that may help us reduce this risk?

I want to keep the mixed mode. There is no harm here I guess. It is not as complicated to use or to understand from a users perspective. It is an additional tool in hand if chunk and delimited do not suffice the needs of the user.
I had also updated code to rectify valgrind leaks. There are some leaks already present in the code mostly due to the use of C++ code from minifi mostly due to cyclic references I believe. I took care of any additional leaks injected by the nanofi tailfile implementation.
I will look into logging once we start the EFM ECU integration implementation. But for now I think this PR is pretty good to be merged.

I can't get any of the ecus actually send flow_files to my NiFi instance.
After a bit of debugging it seems that the uuid the flow_files are written to in the hash table in ecu.c (that is acquired by calling get_proc_uuid_from_context) is not the same uuid the main function tries to read the flow_files from (which is acquired by get_proc_uuid_from_processor).
Please also see my inline comments - some are just suggestions, but there are some leaks and crashes.

Tested all ECUs. I am able to send flow file payload to nifi now. It was a bug with returning .c_str() from a temporary string. Pretty bad I would say.


Code snippet is commented out by mistake

🛑 IMPORTANT: You can get your feedback addressed faster if you use the comment section for the article in which you encountered a problem.
Link to article:
https://docs.microsoft.com/en-us/previous-versions/msp-n-p/dn589799(v%3Dpandp.10)
Problem:
Notice in the Example section, C# code snippet's are completely commented out

@fame Unfortunately, the content is no longer to update due to out of product lifecycle. Please go to the newer version.


Have lorri auto-evalute the first time

When setting up direnv with lorri, it is a bummer to have it fail the first time due to no watch runner. It would be cool if lorri direnv would auto-evaluate the expressions just the first time.
One concern I have is this might encourage people to never watch their project, and thus have ancient dependencies stick around for an eternity.

What if it failed "open" - in other words, if lorri watch isn't running, behave like use nix?

in other words, if lorri watch isn't running, behave like use nix?

We do exactly that in lorri itself:
https://github.com/target/lorri/blob/master/.envrc#L7-L17

I've seen several people replace use nix with a lorri command, so this might not work great in lorri eval, but maybe recommending something like it as a recommended .direnvrc entry?

Closing in favor of https://github.com/target/lorri/issues/99


feat(core): :sparkles: adding new props listType

Description
there was a need of differentiating list to resolve the issue we are adding listType through which user can directly select types of list and use it
Ref: #272
Image

Merging since stale over 24hrs.


Error in packages on windows

https://files.slack.com/files-pri/T01U12D64J3-F0237JDBAM6/image.png

Thanks for contributing @leonardoXoliveira !
Can you share with us:

The steps to reproduce it
Environment you are using (a.k.a WSL, promp, etc)

Thanks!

Hi @tlgimenes, i had the same problem here.
The steps were these: git clone > yarn to install > yarn develop to run.
This error appears in the terminal:
[gatsby-plugin-graphql] Error: Could not find fragment OrderFormFragment_orderForm for query SetRegionMutation
    at QueryManager.ensureRequiredFragments (D:\test\freewaystore.store\node_modules\@vtex\gatsby-plugin-graphql\manager.js:121:23)
    at D:\test\freewaystore.store\node_modules\@vtex\gatsby-plugin-graphql\manager.js:129:18
    at Array.map (<anonymous>)
    at QueryManager.getQueries (D:\test\freewaystore.store\node_modules\@vtex\gatsby-plugin-graphql\manager.js:128:31)
    at D:\test\freewaystore.store\node_modules\@vtex\gatsby-plugin-graphql\webpack.js:123:44
    at Hook.eval [as callAsync] (eval at create (D:\test\freewaystore.store\node_modules\tapable\lib\HookCodeFactory.js:33:10), <anonymous>:37:17)
    at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (D:\test\freewaystore.store\node_modules\tapable\lib\Hook.js:18:14)
    at Watching._done (D:\test\freewaystore.store\node_modules\webpack\lib\Watching.js:199:28)
    at D:\test\freewaystore.store\node_modules\webpack\lib\Watching.js:127:21
    at Compiler.emitRecords (D:\test\freewaystore.store\node_modules\webpack\lib\Compiler.js:846:39)
    at D:\test\freewaystore.store\node_modules\webpack\lib\Watching.js:105:22
    at D:\test\freewaystore.store\node_modules\webpack\lib\Compiler.js:828:14
    at Hook.eval [as callAsync] (eval at create (D:\test\freewaystore.store\node_modules\tapable\lib\HookCodeFactory.js:33:10), <anonymous>:15:1)
    at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (D:\test\freewaystore.store\node_modules\tapable\lib\Hook.js:18:14)
    at D:\test\freewaystore.store\node_modules\webpack\lib\Compiler.js:825:27
    at D:\test\freewaystore.store\node_modules\neo-async\async.js:2818:7

and in localhost, the same error appears in the @leonardoXoliveira print.
I'm using Windows 10 + PowerShell 7.0.3.
I tested in a macOS and just works with yarn if you install with npm there is also an error

Thanks for contributing @milaniromulo!
Currently, we don't support npm, only yarn. Also, this seems like this babel/webpack plugin couldn't find the files to replace. I bet this is something to do with windows file system being non POSIX.
Can you do a test using wsl? This should be a POSIX compliant solution and this could work

Hi @tlgimenes, sorry for the delay. We did tests with WSL and it's works :)
I think we can continue now, tks bro.

Hello @tlgimenes , thanks for clarification.
At wsl, we face some issues with permission and stuff but we managed to solve. We will include this in the documentation if we have questions from other developers.
Tks!

@tlgimenes I think it's better to fix the windows behavior instead of using WSL. Particularly, I don't have a good developing experience using WS, people reported that they had a slow developing experience using WSL and we have a good part of our external devs using Windows OS.
I investigated using Windows OS and we started having those problems after this PR. We are using a process.cwd() there and I found that this function is inconsistent across those platforms, but I'm still reading about (it's from 2016, idk it's deprecated) and I'm comparing OSX and Windows behaviors.
I'm still investigating about this, so I'm reopening this issue.

nice @Gmantiqueira ! I've seem people using program instead of process, maybe this is more consistent. Wanna give it a try? Thanks for taking this issue btw

It seems to be fixed after this update #887. We just need to update our stores.


[Feature] IAM Proposal

IAM is short for Identity and Access Management. We need IAM in hypertrons. @frank-zsy will you share some IAM solutions here?

Sure, you may look into openIAM openLDAP openDJ openAM

We don't have to implement complex IAM solutions. So far, only role component needs unified user ID.  We can config unified userID in hypertrons.json. Unified userID have corresponding account in different hosting platforms. These corresponding accounts are stored in database. Here is an example.
{
	"liwen-tj": {
		"github": liwen-tj,
		"gitlab": liwen,
		"gitee": liwen00812
	}
}

We use github account as default unified userID.

The purposes for investigate IAM system are:

We need a sub system to make sure we can make the correlation of developers on different platform, which means we can save the account or token of developers on different platform so we can more comprehensively analysis the data among platforms.
We need a auth management system, currently we use roles and commands to assign authority to developers, but in a mature system we need a whole sub system or module to handle the authorities and LDAP seems a good solution which provides tree form authority management for different resources rather than just commands.

Although I wish to investigate IAM systems, this does not mean we will need it or it is the best solution, you can look into other solution or open source projects which can solve the problems.


Cannot invoke 'Database' with an argument list of type '(host: String, user: String, password: String, database: String)'

I have migrated my code to Vapor 2.0 and i'm facing this error
Cannot invoke 'Database' with an argument list of type '(host: String, user: String, password: String, database: String)'
extension MySQL.Database {
    static func makeConnection() -> MySQL.Database {
        do {
            let mysql = try MySQL.Database(
                host: "<IP_ADDRESS>",
                user: "root",
                password: "",
                database: "Radar"
            )
            try mysql.execute("SELECT @@version")
            return mysql
        } catch {

            fatalError("MySQL Error")
        }
    }
}

@O-mkar host has been renamed to hostname. This should fix the compiler error


Makes PDF data reading Streams API friendly.

Adds PDFWorkerStream and PDFNetworkStream with classes that can provide readers (that are compatible with ReadableByteStream, see Streams API).
In the future PDFNetworkStream can be re-written to support Fetch API. The PDFWorkerStream can be refactored to be more lightweight and move most of I/O logic to the main thread.

/cc @Rob--W shall help with #5319

Which such a refactor, I'd like to see more unit tests that verifies that the network layer works as intended. That makes it much easier to build upon this interface (e.g. adding support for the streams API).

/botio test

/botio test

/botio-windows preview

@yurydelendik Good job. The interface + documentation made it easier to understand the API design. I've added some comments.
And in the interface all functions are anonymous. Is that intentional? With the similarly named methods in the different interfaces (e.g. read), grepping may be easier if the interface name is included in the function name. I don't mind either way, pick your preference.

/botio test

/botio-linux test

And in the interface all functions are anonymous. Is that intentional? With the similarly named methods in the different interfaces (e.g. read), grepping may be easier if the interface name is included in the function name.

Actually, there is no point on applying function names (it was legacy practice to ensure we are getting proper stack trace reports). IDEs are capable to navigate between code and jsdoc (as well as show pop-up hints). Not sure how can we be more useful here in grep situation, also once we start moving the code to the ES6 we will probably completely abandon this naming conversion in favor of class definition.

/botio-linux preview

/botio test

/botio-linux test

Nice patch!
However, it seems that the issue @timvandermeij mentioned on IRC a few days back is still present in the patch that landed, please see http://logs.glob.uno/?c=mozilla%23pdfjs&s=26+Feb+2016&e=26+Feb+2016#c46715.
For example, open http://ftp.acc.umu.se/mirror/CTAN/info/lshort/english/lshort.pdf#disableRange=true&disableStream=true and check the browser console (Ctrl+Shift+J):
Error: pdfjs/core/network module is not loaded pdf.worker.js:2340:5
assert@resource://pdf.js/build/pdf.worker.js:2356:5
getPdfManager@resource://pdf.js/build/pdf.worker.js:41789:11
wphCreateDocumentHandler/setupDoc@resource://pdf.js/build/pdf.worker.js:41930:7
wphReady@resource://pdf.js/build/pdf.worker.js:42171:7
messageHandlerComObjOnMessage@resource://pdf.js/build/pdf.worker.js:3704:9
EventListener.handleEvent*MessageHandler@resource://pdf.js/build/pdf.worker.js:3710:3
wphCreateDocumentHandler@resource://pdf.js/build/pdf.worker.js:41719:19
wphSetupDoc@resource://pdf.js/build/pdf.worker.js:41706:14
messageHandlerComObjOnMessage/<@resource://pdf.js/build/pdf.worker.js:3681:18
promise callback*messageHandlerComObjOnMessage@resource://pdf.js/build/pdf.worker.js:3680:9
EventListener.handleEvent*MessageHandler@resource://pdf.js/build/pdf.worker.js:3710:3
initializeWorker@resource://pdf.js/build/pdf.worker.js:42218:17
pdfjsWrapper/<@resource://pdf.js/build/pdf.worker.js:42226:3
pdfjsWrapper/<@resource://pdf.js/build/pdf.worker.js:41395:5
pdfjsWrapper@resource://pdf.js/build/pdf.worker.js:41393:2
@resource://pdf.js/build/pdf.worker.js:40:13
@resource://pdf.js/build/pdf.worker.js:25:1
@resource://pdf.js/build/pdf.worker.js:18:2
 pdf.worker.js:2341:5
PDF 47642ae5bebee4e458b77187094e0f92 [1.5 pdfTeX-1.40.14 / LaTeX with hyperref package] (PDF.js: 1.4.109 [WebGL]) viewer.js:7135:7

Hmm, worked for me at http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf#disableAutoFetch=true&disableStream=true

Hmm, worked for me at http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf#disableAutoFetch=true&disableStream=true

In order to trigger this, both disableRange and disableStream need to be set (as in my link above), and you might need to do a skip-cache reload as well.
Another, simpler way to trigger this in Firefox, is to open a local file by dragging it into the browser window (then you don't need to set any disable... parameters).

As outlined in https://github.com/mozilla/pdf.js/pull/6879#issuecomment-191149419, this is not a breaking error. But it'd be nice to get rid of all that console output (caused by the assert), so couldn't we just do something like the diff below?
diff --git a/src/core/worker.js b/src/core/worker.js
index c78b52e..4e4446e 100644
--- a/src/core/worker.js
+++ b/src/core/worker.js
@@ -548,8 +548,14 @@ var WorkerMessageHandler = PDFJS.WorkerMessageHandler = {
         if (source.chunkedViewerLoading) {
           pdfStream = new PDFWorkerStream(source, handler);
         } else {
+//#if (FIREFOX || MOZCENTRAL)
+//        pdfManagerCapability.reject(new Error('PDFNetworkStream is ' +
+//          'undefined in the addon/built-in version.'));
+//        return pdfManagerCapability.promise;
+//#else
           assert(PDFNetworkStream, 'pdfjs/core/network module is not loaded');
           pdfStream = new PDFNetworkStream(data);
+//#endif
         }
       } catch (ex) {
         pdfManagerCapability.reject(ex);

As outlined in #6879 (comment), this is not a breaking error. But it'd be nice to get rid of all that console output (caused by the assert), so couldn't we just do something like the diff below?

Problem that it shall not be called without source.chunkedViewerLoading set in add-on. I'm trying to understand how it's happening.

It's due to this case https://github.com/mozilla/pdf.js/blob/master/web/viewer.js#L512 and we already resolved a promise at https://github.com/mozilla/pdf.js/blob/master/src/core/worker.js#L537 for that (that's why it is working).


Support Taproot

Taproot is a next major Bitcoin softfork.  It is expected to get activated on mainnet in mid-November.
Tasks:

[x] Support sending to Taproot addresses (should not be enabled before receive support in popular wallets)  (mostly done)
[ ] Schnorr signature support
[ ] Taproot primitives in scripts

/later:/

[ ] Create Taproot address, support sending from Taproot address (receive)

Docs:

BIP 340: Schnorr Signatures
BIP 341: Taproot: SegWit version 1 spending rules
BIP 342: Validation of Taproot Scripts
BIP 350: Bech32m format for v1+ witness addresses

Segwit v1 support, with Bech32m encoding is already supported, see:
https://github.com/trustwallet/wallet-core/pull/1345
https://community.trustwallet.com/t/taproot-support-send-already-supported/213127

Schnorr support in the included Trezor library:

A schnorr version is included (from longer time), bot used

A wc-specific variation exists for Zilliqa

Current latest Trezor contains Bitcoin ABC Schnorr, which is a different variant

BTC Schnorr support is work-in-progress on in Trezor, see:

https://github.com/trezor/trezor-firmware/issues/1534  Issue: Re-enable secp256k1-zkp and make it default

https://github.com/trezor/trezor-firmware/pull/1678  PR

https://github.com/trezor/trezor-firmware/issues/1656  Issue: Support Taproot

We should include Trezor version once it is availble.

0xa36fD8fF0524f287422337d99bABdf05923f581B


Attaching with SSH transport and GDB Native debugging does not always show Modules loaded

VS Version: 2017 RC (2 or 3)
Rerpo:

On a linux machine, create a simple C++ executable that calls scanf (or something similar)
Build it and run it on linux.
From VS 2017 RC, attach using the SSH Transport (Debug -> Attach to process, change 'Connection Type' to SSH)
Once attached, view the modules window

Expected: The modules window should show at least two modules
Actual: The modules window may show no modules (even though the library-loaded events appear in the MIEngine log).
This appears to be a race as it doesn't repro consistently. I can see the events in the MI log even when I don't see modules loaded in the modules window.

Full log:
1: (218) ->=thread-group-added,id="i1"
1: (219) ->~"GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1\n"
1: (219) ->~"Copyright (C) 2016 Free Software Foundation, Inc.\n"
1: (219) ->~"License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.  Type \"show copying\"\nand \"show warranty\" for details.\n"
1: (219) ->~"This GDB was configured as \"x86_64-linux-gnu\".\nType \"show configuration\" for configuration details."
1: (219) ->~"\nFor bug reporting instructions, please see:\n"
1: (219) ->~"<http://www.gnu.org/software/gdb/bugs/>.\n"
1: (219) ->~"Find the GDB manual and other documentation resources online at:\n<http://www.gnu.org/software/gdb/documentation/>.\n"
1: (219) ->~"For help, type \"help\".\n"
1: (219) ->~"Type \"apropos word\" to search for commands related to \"word\".\n"
1: (219) ->(gdb)
=thread-group-added,id="i1"
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
1: (228) <-1001-gdb-set target-async on
1: (229) ->1001^done
1: (229) ->(gdb)
1: (230) 1001: elapsed time 1
1: (1046) <-1002-gdb-set stop-on-solib-events 1
1: (1143) ->1002^done
1: (1143) ->(gdb)
1: (1143) 1002: elapsed time 96
1: (1151) <-1003-interpreter-exec console "shell readlink -f /proc/9023/exe"
1: (1155) ->/home/chuckr/source/scratch2/a.out
1: (1155) ->1003^done
1: (1155) ->(gdb)
1: (1155) 1003: elapsed time 3
1: (1157) <-1004-file-exec-and-symbols /home/chuckr/source/scratch2/a.out
1: (1159) ->1004^done
1: (1159) ->(gdb)
1: (1159) 1004: elapsed time 1
1: (1159) <-1005-interpreter-exec console "show architecture"
1: (1160) ->~"The target architecture is set automatically (currently i386:x86-64)\n"
1: (1160) ->1005^done
1: (1160) ->(gdb)
1: (1160) 1005: elapsed time 0
1: (1162) <-1006-target-attach 9023
1: (1163) ->=thread-group-started,id="i1",pid="9023"
1: (1163) ->=thread-created,id="1",group-id="i1"
1: (1163) ->1006^done
1: (1165) ->=library-loaded,id="/lib/x86_64-linux-gnu/libc.so.6",target-name="/lib/x86_64-linux-gnu/libc.so.6",host-name="/lib/x86_64-linux-gnu/libc.so.6",symbols-loaded="0",thread-group="i1"
1: (1165) ->=library-loaded,id="/lib64/ld-linux-x86-64.so.2",target-name="/lib64/ld-linux-x86-64.so.2",host-name="/lib64/ld-linux-x86-64.so.2",symbols-loaded="0",thread-group="i1"
1: (1169) 1006: elapsed time 6
0x00007f37392ff680 in __read_nocancel () at ../sysdeps/unix/syscall-template.S:84
1: (1238) ->~"0x00007f37392ff680 in __read_nocancel () at ../sysdeps/unix/syscall-template.S:84\n"
1: (1239) ->&"84\t../sysdeps/unix/syscall-template.S: No such file or directory.\n"
1: (1239) ->*stopped,frame={addr="0x00007f37392ff680",func="__read_nocancel",args=[],file="../sysdeps/unix/syscall-template.S",fullname="/build/glibc-Qz8a69/glibc-2.23/io/../sysdeps/unix/syscall-template.S",line="84"},thread-id="1",stopped-threads="all",core="7"
1: (1266) <-1007-thread-info
1: (1267) ->1007^done,threads=[{id="1",target-id="process 9023",name="a.out",frame={level="0",addr="0x00007f37392ff680",func="__read_nocancel",args=[],file="../sysdeps/unix/syscall-template.S",fullname="/build/glibc-Qz8a69/glibc-2.23/io/../sysdeps/unix/syscall-template.S",line="84"},state="stopped",core="7"}],current-thread-id="1"
1: (1268) ->(gdb)
1: (1271) 1007: elapsed time 5
1: (1288) <-1008-stack-list-frames 0 1000
1: (1294) ->1008^done,stack=[frame={level="0",addr="0x00007f37392ff680",func="__read_nocancel",file="../sysdeps/unix/syscall-template.S",fullname="/build/glibc-Qz8a69/glibc-2.23/io/../sysdeps/unix/syscall-template.S",line="84"},frame={level="1",addr="0x00007f37392835e8",func="_IO_new_file_underflow",file="fileops.c",fullname="/build/glibc-Qz8a69/glibc-2.23/libio/fileops.c",line="592"},frame={level="2",addr="0x00007f373928460e",func="__GI__IO_default_uflow",file="genops.c",fullname="/build/glibc-Qz8a69/glibc-2.23/libio/genops.c",line="413"},frame={level="3",addr="0x00007f3739265260",func="_IO_vfscanf_internal",file="vfscanf.c",fullname="/build/glibc-Qz8a69/glibc-2.23/stdio-common/vfscanf.c",line="634"},frame={level="4",addr="0x00007f373927387b",func="__scanf",file="scanf.c",fullname="/build/glibc-Qz8a69/glibc-2.23/stdio-common/scanf.c",line="33"},frame={level="5",addr="0x00000000004005c3",func="main",file="main.cpp",fullname="/home/chuckr/source/scratch2/main.cpp",line="6"}]
1: (1294) ->(gdb)
1: (1294) 1008: elapsed time 5
1: (1301) <--exec-continue
1: (1302) ->^running
1: (1303) ->*running,thread-id="all"
1: (1303) ->(gdb)

I'm sorry but may I ask how you did to attach it?
I tried to do the same thing as you, but got an error that "Unable to start debugging. Unable to establish a connection to GDB. Debug output may contain more information."
It works well when I just write code and use the remote GDB.

@jcf94 what does the output window say? If it isn't obvious from that, please open a new issue.


Add support for containing class fields to RxNullabilityPropagator

Class field information is propagated from filter to map in RxNullabilityPropagator
Fixes #282
Added unit test

What's the rationale again for this?
One potential issue I see with this change, is that, while it is likely to work for filter(...).map(...) it's aggressively thread unsafe and dangerous in a more general case.
Consider: filter(s -> ...).delay(1000).map(s -> ...)
It's safe to assume the values of fields rooted at s haven't changed in between the end of filter and the start of map, since they are "internal" to the stream that got delayed. However, if those lambdas access fields of the containing class, those can easily change between those two program points.
We do something similar for captured locals from the enclosing context (see here), but, in that case, it's actually safe since only effectively final locals from enclosing contexts can be accessed within a lambda. With class fields, this is not true at all, and mutable fields can be accessed, leading to the problem above.
NullAway doesn't attempt to be fully sound with respect to concurrency, but this seems like the kind of optimistic treatment that can easily cause us to miss bugs in the future, so I'd want to make sure there is enough value in supporting this case.
Thoughts, @msridhar ?

What's the rationale again for this?

@lazaroclapp, to be fair you did open #282 😄
That said, I agree with your concern regarding soundness when successive operators are accessing mutable state.  The developer can "fix" the example in #282 by first storing foo in an effectively-final local l, and then accessing l in the callbacks.
So maybe we should say #282 is a 'wontfix'?  @shas19 thoughts?

What's the rationale again for this?

@lazaroclapp, to be fair you did open #282 😄

Heh. Fair. My bad! I didn't remember that.
I still think it's fixable, but might need distinguishing at which point the stream might be interrupted and stuff like that. It just bugs me either way :/

That said, I agree with your concern regarding soundness when successive operators are accessing mutable state. The developer can "fix" the example in #282 by first storing foo in an effectively-final local l, and then accessing l in the callbacks.

That would work, but is also remarkably non-intuitive. Might be the best we can do, though.

So maybe we should say #282 is a 'wontfix'? @shas19 thoughts?

Ohh... The problem seems harder than I thought, definitely this is not enough.

That said, I agree with your concern regarding soundness when successive operators are accessing mutable state. The developer can "fix" the example in #282 by first storing foo in an effectively-final local l, and then accessing l in the callbacks.

That would work, but is also remarkably non-intuitive. Might be the best we can do, though.

It is non-intuitive, but it's also how Java works.  I think this kind of thing is important to understand for an experienced Java programmer.  Sometimes you need to do this manual extracting of locals in perf-critical code too, when the JIT isn't smart enough to do redundant load elimination.  Kotlin has the nice variable?.let syntax to make this easier (see here), but we are stuck with Java.


chore(android): Remove outdated appDistribution group

Merging this because the Android release build is broken without it.


[pinot-spark-connector] Fix empty data table handling in GRPC reader

Fixes a bug where an exception is thrown when all of the data tables retuned in a Spark 'split' is empty.
When the Spark reader pushes down a filter which effectively reduces the returned data for all the segments to zero, GRPC data fetcher (PinotGrpcServerDataFetcher) wrongly assumes that there was an error. This behavior was introduced when porting logic from the non-GRPC fetcher (PinotServerDataFetcher) where the assumption is correct. Http and GRPC interfaces differ in behavior where the HTTP interface will return empty instances of datatables whereas with GRPC you only get metadata.
I'm updating the error handling to remove this assumption and record an error when there is actual exception thrown by the GRPC server.
bugfix

Codecov Report

Merging #9837 (abc8c65) into master (3724ba2) will decrease coverage by 18.87%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #9837       +/-   ##
=============================================
- Coverage     34.69%   15.82%   -18.88%
+ Complexity      190      175       -15
=============================================
  Files          1965     1912       -53
  Lines        105115   102694     -2421
  Branches      15909    15621      -288
=============================================
- Hits          36474    16251    -20223
- Misses        65542    85248    +19706
+ Partials       3099     1195     -1904

Flag
Coverage Δ

integration1
?

unittests2
15.82% <ø> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...src/main/java/org/apache/pinot/sql/FilterKind.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ain/java/org/apache/pinot/core/data/table/Key.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...n/java/org/apache/pinot/core/data/table/Table.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/core/data/table/Record.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/core/util/GroupByUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/common/auth/AuthConfig.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...va/org/apache/pinot/core/routing/RoutingTable.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/sql/parsers/PinotSqlType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...a/org/apache/pinot/core/operator/BaseOperator.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...va/org/apache/pinot/common/config/NettyConfig.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 674 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Thanks @Jackie-Jiang !


Android: desugar fails for Reference2DoubleMap

Android build toolchain fails to desugar 0.8.3:
> Task :app:transformClassesWithDexBuilderForDevDebug
AGPBI: {"kind":"error","text":"One or more instruction is preventing default interface method from being desugared: java.lang.Double it.unimi.dsi.fastutil.objects.Reference2DoubleMap.getOrDefault(java.lang.Object, java.lang.Double)","sources":[{"file":"/Users/astar/.gradle/caches/modules-2/files-2.1/it.unimi.dsi/fastutil/8.3.0/742307990505e3a149c9c60825ffc1db5ceef02e/fastutil-8.3.0.jar"}],"tool":"D8"}
AGPBI: {"kind":"error","text":"One or more instruction is preventing default interface method from being desugared: java.util.Comparator it.unimi.dsi.fastutil.floats.FloatComparator.thenComparing(java.util.Comparator)","sources":[{"file":"/Users/astar/.gradle/caches/modules-2/files-2.1/it.unimi.dsi/fastutil/8.3.0/742307990505e3a149c9c60825ffc1db5ceef02e/fastutil-8.3.0.jar"}],"tool":"D8"}
org.gradle.workers.WorkerExecutionException: There were multiple failures while executing work items
	at org.gradle.workers.internal.DefaultWorkerExecutor.workerExecutionException(DefaultWorkerExecutor.java:301)
	at org.gradle.workers.internal.DefaultWorkerExecutor.await(DefaultWorkerExecutor.java:276)
...
Caused by: com.android.tools.r8.utils.AbortException: Error: /Users/astar/.gradle/caches/modules-2/files-2.1/it.unimi.dsi/fastutil/8.3.0/742307990505e3a149c9c60825ffc1db5ceef02e/fastutil-8.3.0.jar:it/unimi/dsi/fastutil/floats/FloatComparator.class, One or more instruction is preventing default interface method from being desugared: java.util.Comparator it.unimi.dsi.fastutil.floats.FloatComparator.thenComparing(java.util.Comparator)
	at com.android.tools.r8.utils.Reporter.a(:21)
	at com.android.tools.r8.utils.Reporter.a(:7)
	at com.android.tools.r8.utils.t.a(:36)
	... 36 more

OMG. I have literally no idea of what this is. Can you elaborate?

when fastutil 8.* added to android project with
compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }

there is special step during build to "desugar" java 8 code. So java 8 features would work on older jvm android is limited to.
Here is info about Android Java 8 support: https://developer.android.com/studio/write/java8-support
And all those lambdas/default interface methods/etc usually "just work".
Not sure if it's "too magic" on fastutil side with this specific default interface methods implementations or D8 issue.

I'd try to pass this into the maintainers of the "desugar" code—fastutil is just compiled with target=1.8...

same issue here since gradle update to 3.5.1

any update on this issue? I got stuck into this issue since I upgrade the android studio to 3.6. The java version remains at 1.8.

Still valid for 8.3.1 with:
classpath 'com.android.tools.build:gradle:3.6.3'
distributionUrl=https\://services.gradle.org/distributions/gradle-6.1.1-all.zip

One or more instruction is preventing default interface method from being desugared: java.lang.Double it.unimi.dsi.fastutil.objects.Reference2DoubleMap.getOrDefault(java.lang.Object, java.lang.Double)

Once again, I totally have no idea :(.

Yeah, but I do not see any issues created for d8/r8 team referenced here.

I'd try to pass this into the maintainers of the "desugar" code—fastutil is just compiled with

A D8 issue has been filed for this and should be resolved soon: https://issuetracker.google.com/issues/157806261
FYI, the issue is a failure to rewrite the call from default method override to the default method in the library, in this case the default interface method CharConsumer.andThen forwarding to the default library method Consumer.andThen.


pycuda -> bifrost question

Next question / feature request: getting data from a pycuda array into a bifrost ndarray. I replaced all instances of np.ndarray with bf.ndarray in ring2.py and it so far seems to work. This means that I can write things like  ispans[0].data.as_GPUArray(), which is great.
How would I do the opposite, and get data efficiently from a pycuda gpuarray into an output ring buffer? Here's my code to do a batched FFT with scikit-cuda and pycuda:
class GpuSpec(bfp.TransformBlock):
    def __init__(self, iring, space, *args, **kwargs):
        super(GpuSpec, self).__init__([iring], *args, **kwargs)

        self.fft_plan_init = False
        self.df_gpu_init   = False
        self.d_gpu_init    = False

    def on_sequence(self, iseqs):
        ohdr = deepcopy(iseqs[0].header)
        # Tensor shape is (sequence, pol, coarse channel, time)
        ohdr['_tensor']['dtype'] = 'cf32'
        ohdr['_tensor']['shape'] = [-1, 2, 64, 524288]
        return [ohdr], [None]

    @timeit
    def on_data(self, ispans, ospans):
        self.context = pycuda.tools.make_default_context()
        self.device = self.context.get_device()

        d_x = ispans[0].data[0, 0].as_GPUArray()
        d_y = ispans[0].data[0, 1].as_GPUArray()

        if not self.fft_plan_init:
            self.fft_plan_init = True
            #               Plan(N_fft,        input_dtype,  output_dtype, batch_size)
            self.fft_plan = Plan(d_x.shape[1], np.complex64, np.complex64, d_x.shape[0])

        if not self.df_gpu_init:
            self.df_gpu_init = True
            self.df_gpu_x  = gpuarray.empty((d_x.shape[0], d_x.shape[1]), np.complex64)
            self.df_gpu_y  = gpuarray.empty((d_y.shape[0], d_y.shape[1]), np.complex64)

        ## Do FFT
        cufft(d_x, self.df_gpu_x, self.fft_plan)
        cufft(d_y, self.df_gpu_y, self.fft_plan)

        ospans[0].data[0, 0] = self.df_gpu_x.get()
        ospans[0].data[0, 1] = self.df_gpu_y.get()

        self.context.pop()
        return [1]

The final .get() functions are making this execute slowly as it's presumably copying to system then back into the output array.
The self.context and self.device lines are only in there to get things to work. Is there a better way to do this?

Import FYI:
try:
    import pycuda.driver as cuda
    import pycuda.autoinit
    import pycuda.gpuarray as gpuarray
    import pycuda.cumath as cumath
    import pycuda.tools
except ImportError:
    raise RuntimeError("could not load pycuda. Please check your install.")

try:
    from skcuda.fft import Plan
    from skcuda.fft import fft as cufft
    from skcuda.misc import multiply as cumultiply
except ImportError:
    raise RuntimeError("Could not load scikit-cuda. Please check your install")

bf.asarray(GPUArray_object) will create a Bifrost ndarray with space=cuda
that wraps the GPU memory owned by the GPUArray.
bf.ndarray.as_GPUArray() will copy the ndarray (of any space) into a new
GPUArray.
On 22 January 2017 at 16:06, Danny Price<EMAIL_ADDRESS>wrote:

Import FYI:
try:
import pycuda.driver as cuda
import pycuda.autoinit
import pycuda.gpuarray as gpuarray
import pycuda.cumath as cumath
import pycuda.toolsexcept ImportError:
raise RuntimeError("could not load pycuda. Please check your install.")
try:
from skcuda.fft import Plan
from skcuda.fft import fft as cufft
from skcuda.misc import multiply as cumultiplyexcept ImportError:
raise RuntimeError("Could not load scikit-cuda. Please check your install")
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ledatelescope/bifrost/issues/66#issuecomment-274371608,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADy3WDmY7U5r81vS03m3AFEn5NoL_0Jwks5rU-7ygaJpZM4Lqjrl
.

@benbarsdell that is freakin' awesome!
Now I'm running in to a segfault:
a = np.random.random(1000)
b = gpuarray.to_gpu(a)
c = bf.asarray(b)
d = bf.ndarray(np.zeros(2000))
d[:1000] = c

causes:
[bldcpr:28060] *** Process received signal ***
[bldcpr:28060] Signal: Segmentation fault (11)
[bldcpr:28060] Signal code: Invalid permissions (2)
[bldcpr:28060] Failing at address: 0x1020b000000
[bldcpr:28060] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0)[0x7f6de3f393d0]
[bldcpr:28060] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x14d4a6)[0x7f6de3cac4a6]
[bldcpr:28060] [ 2] /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so(+0x6be48)[0x7f6ddecf3e48]
[bldcpr:28060] [ 3] /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so(+0x6c75a)[0x7f6ddecf475a]
[bldcpr:28060] [ 4] /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so(+0x21660)[0x7f6ddeca9660]
[bldcpr:28060] [ 5] /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so(+0x1362c0)[0x7f6ddedbe2c0]
[bldcpr:28060] [ 6] /usr/bin/python[0x5076ce]
[bldcpr:28060] [ 7] /usr/bin/python(PyEval_EvalFrameEx+0x3834)[0x4c7834]
[bldcpr:28060] [ 8] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [ 9] /usr/bin/python(PyEval_EvalFrameEx+0x44db)[0x4c84db]
[bldcpr:28060] [10] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [11] /usr/bin/python(PyEval_EvalFrameEx+0x6099)[0x4ca099]
[bldcpr:28060] [12] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [13] /usr/bin/python(PyEval_EvalFrameEx+0x6099)[0x4ca099]
[bldcpr:28060] [14] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [15] /usr/bin/python(PyEval_EvalFrameEx+0x6099)[0x4ca099]
[bldcpr:28060] [16] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [17] /usr/bin/python(PyEval_EvalFrameEx+0x6099)[0x4ca099]
[bldcpr:28060] [18] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [19] /usr/bin/python(PyEval_EvalFrameEx+0x6099)[0x4ca099]
[bldcpr:28060] [20] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [21] /usr/bin/python(PyEval_EvalFrameEx+0x68d1)[0x4ca8d1]
[bldcpr:28060] [22] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [23] /usr/bin/python[0x4de8b8]
[bldcpr:28060] [24] /usr/bin/python(PyObject_Call+0x43)[0x4b0cb3]
[bldcpr:28060] [25] /usr/bin/python(PyEval_EvalFrameEx+0x2ad1)[0x4c6ad1]
[bldcpr:28060] [26] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [27] /usr/bin/python(PyEval_EvalFrameEx+0x6099)[0x4ca099]
[bldcpr:28060] [28] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [29] /usr/bin/python(PyEval_EvalCode+0x19)[0x4c2509]
[bldcpr:28060] *** End of error message ***
Segmentation fault (core dumped)

Yes unfortunately assignment like this doesn't work at the moment. It
requires a custom implementation, which is doable but tricky to make
flexible (e.g., to allow dtype conversion and shape broadcasting). Right
now it just lets Numpy deal with it, which results in a segfault if
space=cuda because Numpy tries to memcpy the GPU data.
Actually something that would be cool to try (but is completely untested)
is using space="cuda_managed" under CUDA 8.0.
On 22 January 2017 at 19:00, Danny Price<EMAIL_ADDRESS>wrote:

@benbarsdell https://github.com/benbarsdell that is freakin' awesome!
Now I'm running in to a segfault:
a = np.random.random(1000)
b = gpuarray.to_gpu(a)
c = bf.asarray(b)
d = bf.ndarray(np.zeros(2000))
d[:1000] = c
causes:
[bldcpr:28060] *** Process received signal ***
[bldcpr:28060] Signal: Segmentation fault (11)
[bldcpr:28060] Signal code: Invalid permissions (2)
[bldcpr:28060] Failing at address: 0x1020b000000
[bldcpr:28060] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0)[0x7f6de3f393d0]
[bldcpr:28060] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x14d4a6)[0x7f6de3cac4a6]
[bldcpr:28060] [ 2] /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so(+0x6be48)[0x7f6ddecf3e48]
[bldcpr:28060] [ 3] /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so(+0x6c75a)[0x7f6ddecf475a]
[bldcpr:28060] [ 4] /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so(+0x21660)[0x7f6ddeca9660]
[bldcpr:28060] [ 5] /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so(+0x1362c0)[0x7f6ddedbe2c0]
[bldcpr:28060] [ 6] /usr/bin/python[0x5076ce]
[bldcpr:28060] [ 7] /usr/bin/python(PyEval_EvalFrameEx+0x3834)[0x4c7834]
[bldcpr:28060] [ 8] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [ 9] /usr/bin/python(PyEval_EvalFrameEx+0x44db)[0x4c84db]
[bldcpr:28060] [10] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [11] /usr/bin/python(PyEval_EvalFrameEx+0x6099)[0x4ca099]
[bldcpr:28060] [12] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [13] /usr/bin/python(PyEval_EvalFrameEx+0x6099)[0x4ca099]
[bldcpr:28060] [14] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [15] /usr/bin/python(PyEval_EvalFrameEx+0x6099)[0x4ca099]
[bldcpr:28060] [16] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [17] /usr/bin/python(PyEval_EvalFrameEx+0x6099)[0x4ca099]
[bldcpr:28060] [18] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [19] /usr/bin/python(PyEval_EvalFrameEx+0x6099)[0x4ca099]
[bldcpr:28060] [20] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [21] /usr/bin/python(PyEval_EvalFrameEx+0x68d1)[0x4ca8d1]
[bldcpr:28060] [22] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [23] /usr/bin/python[0x4de8b8]
[bldcpr:28060] [24] /usr/bin/python(PyObject_Call+0x43)[0x4b0cb3]
[bldcpr:28060] [25] /usr/bin/python(PyEval_EvalFrameEx+0x2ad1)[0x4c6ad1]
[bldcpr:28060] [26] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [27] /usr/bin/python(PyEval_EvalFrameEx+0x6099)[0x4ca099]
[bldcpr:28060] [28] /usr/bin/python(PyEval_EvalCodeEx+0x255)[0x4c2765]
[bldcpr:28060] [29] /usr/bin/python(PyEval_EvalCode+0x19)[0x4c2509]
[bldcpr:28060] *** End of error message ***
Segmentation fault (core dumped)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ledatelescope/bifrost/issues/66#issuecomment-274387749,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADy3WMVArXEutHRvqCe4bv99Ui9zM7Xsks5rVBfHgaJpZM4Lqjrl
.

Hrrrm, ok, I see that is a big can of worms. In which case, is there any way I can get pycuda gpuarray data into a bifrost ring buffer on the GPU?
In particular, the case where the dimensions match, e.g. ospans[0].data[0] = bf_array would be very useful.
I guess the other option is to bite the bullet and write code in C++ to directly write into the ring buffer...

Closing this for now as mainly answered


Feature Request: Slug override

Possibility to override post slug (and destination).
For example IMDb movie reviews, you could specify "slug: /reviews/movies/0379217" for review of Coffee and Cigarettes -review on your site.
Other use cases: automatically generated entries, twitter timeline mirroring ("slug: /twitter/[tweet id]").
You could check if the output folder has the entry.

Interesting idea. Keep in mind that the way the document root is structured, the only ways to do this would be:

Add a Redirect to .htaccess for every custom slug, which is probably not a good idea.
Actually generate the folders and files to correspond to the custom slugs. So there would actually be a file in the document root at /reviews/movies/0379217, with all of the parent folders created as well.

I don't love either solution, but I guess the second would probably have fewer problems.

The second solution would be more like what I had in mind.
It would be important to be able to check does a specific entry exist. Wouldn't want to overwrite changes.
Current output naming scheme prevents reliable testing.


[ANCHOR-212] Skip setting amountExpected when posting SEP24 transactions

PR Checklist
PR Structure

[ ] This PR has reasonably narrow scope (if not, break it down into smaller PRs).
[ ] This PR avoids mixing refactoring changes with feature changes (split into two PRs
otherwise).
[ ] This PR's title starts with name of package that is most changed in the PR, ex.
paymentservice.stellar, or all or doc if the changes are broad or impact many
packages.

Thoroughness

[ ] This PR adds tests for the most critical parts of the new functionality or fixes.

What
Skip setting amountExpected when posting SEP24 transactions
Why
The amountExpected should be set by the business server through the PATCH /transaction API.

@lijamie98 is it still set in JWT token?
FYI: it should be always set, if it's present in the request (regardless of configuration)

Reference Server Preview is available here:https://anchor-ref-pr833.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr833.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr833.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr833.previews.kube001.services.stellar-ops.com/

The JWT has a data object that should have any fields passed in the POST /transactions/deposit/interactive or POST /transactions/withdraw/interactive request bodies that shouldn't be persisted in the anchor platform's database. Right now this includes any of the SEP-9 KYC fields, but it could also include the amount field.
I think if we're not saving it to amountExpected automatically, we need to pass it to the business somehow, and using the JWT makes sense.

@JakeUrban I think this make sense now. Is amount the only field that we should add to JWT? If there other fields in the request that are not saved in database but should be passed to JWT, we can add them all at once.
Here is the list that I can think of. Please comment:

transaction_id
all SEP-9 fields
amount

Reference Server Preview is available here:https://anchor-ref-pr833.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr833.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr833.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr833.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr833.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr833.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr833.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr833.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr833.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr833.previews.kube001.services.stellar-ops.com/


Command exit status:   143

Hi Jackson Lab,
I was running your cs-nf-GBRS pipeline and was were able to have it work over one sample but when we tried to run this over multiple samples, we run into this error.
Caused by:
Process GBRS:RUN_EMASE:GBRS_BAM2EMASE (S3) terminated with an error exit status (143)
Command executed:
gbrs bam2emase -i S3_mapped_R2.bam                 -m /path/to/output/emase.fullTranscripts.info                 -h A,B,C,D,E,F,G,H                 -o S3_mapped_R2.emase.h5
Command exit status:
143
We have also been seeing this error as well with exit status 143
Command executed:
set -o pipefail
zcat S3_R1_001.fastq.gz
| bowtie -p 8 -q -a --best --strata --sam -v 3 -x /path/to/output/bowtie.transcripts - 2> S3.bowtie_R1.log
| samtools view -bS - > S3_mapped_R1.bam
Command exit status:
143
Do you know what may be causing this issue?

Exit status 143 typically occurs either when the memory request, or the wallclock request has been exceeded.
Can you provide the number of reads in S3_R1_001.fastq.gz?
The second thing to check would be to look in the directory that should be printed to the end of the Nextflow log:
For example:
Command executed:

  Rscript my_script.R

Command exit status:
  143

Command output:
  ...text....

Command error:
 ...text...

Work dir:
  /scratch/$USER/workdir/eb/33d560d5f3e2ee78d5db43ba072228

You would navigate to: /scratch/$USER/workdir/eb/33d560d5f3e2ee78d5db43ba072228 and look at the .command.log file. What (if anything) is appended to the end of that log file? E.g., if you are on slurm you would likely see some epilogue information (memory usage, wall clock usage, etc.). Does that log report memory usage that exceeds the request (200GB).
This step GBRS_BAM2EMASE is known to be memory intensive. To fix this issue, increase the memory allocation in the modules/gbrs/gbrs_bam2emase.nf file beyond 200GB.

@lawtyunc were you successful in troubleshooting this issue?

Yes we were able to fix this but now we are running into this docker image issue.
Caused by:
Failed to pull singularity image
command: singularity pull  --name broadinstitute-gatk-<IP_ADDRESS>.img.pulling.1722359365671 docker://broadinstitute/gatk:<IP_ADDRESS> > /dev/null
status : 143

From: Mike Lloyd @.>
Sent: Tuesday, July 30, 2024 3:19 PM
To: TheJacksonLaboratory/cs-nf-pipelines @.>
Cc: Laws, Tyler @.>; Mention @.>
Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)
You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification
@lawtyunchttps://github.com/lawtyunc were you successful in troubleshooting this issue?
—
Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2259042727, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MSYGOWQ2XXBKFAVNHXDZO7RKXAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJZGA2DENZSG4.
You are receiving this because you were mentioned.Message ID: @.***>

The fact that this fails during the command: singularity pull step along with status: 143 suggests that you are not providing enough memory to the Nextflow job, or that you do not have enough memory on the node/system you are trying to run the Nextflow command on.
Nextflow builds and caches containers for modules as needed. The build/pull is done by Nextflow, and isn't captured by the specific memory reservations for each compute step. So however much memory you have on the node where nextflow run ... is being run, is the memory limit provided to singularity pull ...
I would suggest increasing the memory request to your main nextflow run ... job and see if this issue resolves.
Unfortunately, beyond that recommendation, I am unable to troubleshoot system/HPC specific issues on systems external to Jax. There are too many possible configuration and options to understand the specifics in such cases.
Given that your initial issue is resolved, I will close this ticket, but please feel free to open a new issue if other pipeline related things come up.

Also one question I had about the GBRS pipeline. Based on the outputs do you know what files or references you could use to make a kinship matrix?

From: Mike Lloyd @.>
Sent: Tuesday, July 30, 2024 3:47 PM
To: TheJacksonLaboratory/cs-nf-pipelines @.>
Cc: Laws, Tyler @.>; Mention @.>
Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)
You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification
The fact that this fails during the command: singularity pull step along with status: 143 suggests that you are not providing enough memory to the Nextflow job, or that you do not have enough memory on the node/system you are trying to run the Nextflow command on.
Nextflow builds and caches containers for modules as needed. The build/pull is done by Nextflow, and isn't captured by the specific memory reservations for each compute step. So however much memory you have on the node where nextflow run ... is being run, is the memory limit provided to singularity pull ...
I would suggest increasing the memory request to your main nextflow run ... job and see if this issue resolves.
Unfortunately, beyond that recommendation, I am unable to troubleshoot system/HPC specific issues on systems external to Jax. There are too many possible configuration and options to understand the specifics in such cases.
Given that your initial issue is resolved, I will close this ticket, but please feel free to open a new issue if other pipeline related things come up.
—
Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2259090444, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MS5AF6DKJQMAWQD2W2LZO7UWFAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJZGA4TANBUGQ.
You are receiving this because you were mentioned.Message ID: @.***>

I would suggest the genoprobs object that is output from GBRS, and perhaps the QTL2 package.

Hey Mike,
I thank you for the suggestion. That is actually the package that I found and have been looking into as well. I was wondering. Do you know how this would work for the control data they mention as an input or how that would be generated?

From: Mike Lloyd @.>
Sent: Monday, August 5, 2024 10:15 AM
To: TheJacksonLaboratory/cs-nf-pipelines @.>
Cc: Laws, Tyler @.>; Mention @.>
Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)
I would suggest the genoprobs object that is output from GBRS, and perhaps the QTL2https://kbroman.org/qtl2/ package.
—
Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2269186567, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MS7G3MCEDZVHFLHJZSTZP6CHVAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRZGE4DMNJWG4.
You are receiving this because you were mentioned.Message ID: @.***>

Also when  you say Genoprobs object are you talking about the output npz file?
[cid:64ceb246-7c09-4f08-bf7a-23d6b778632b]

From: Laws, Tyler @.>
Sent: Monday, August 5, 2024 10:25 AM
To: TheJacksonLaboratory/cs-nf-pipelines @.>
Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)
Hey Mike,
I thank you for the suggestion. That is actually the package that I found and have been looking into as well. I was wondering. Do you know how this would work for the control data they mention as an input or how that would be generated?

From: Mike Lloyd @.>
Sent: Monday, August 5, 2024 10:15 AM
To: TheJacksonLaboratory/cs-nf-pipelines @.>
Cc: Laws, Tyler @.>; Mention @.>
Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)
I would suggest the genoprobs object that is output from GBRS, and perhaps the QTL2https://kbroman.org/qtl2/ package.
—
Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2269186567, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MS7G3MCEDZVHFLHJZSTZP6CHVAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRZGE4DMNJWG4.
You are receiving this because you were mentioned.Message ID: @.***>

per the wiki: https://github.com/TheJacksonLaboratory/cs-nf-pipelines/wiki/GBRS-Pipeline-ReadMe
*/gbrs/*.interpolated.genoprobs.tsv: Interpolated genotype probabilities in tab delimited format
Beyond this, I am not able to help with your analysis.

Hey Mike,
Another question I had was how did you guys create the  "/Mus_musculus.GRCm38.dna.toplevel.primaryChr.contig_list" file
for the WGS pipeline. Also would you happen to have this available for GRCm39?

From: Mike Lloyd @.>
Sent: Tuesday, August 6, 2024 11:11 AM
To: TheJacksonLaboratory/cs-nf-pipelines @.>
Cc: Laws, Tyler @.>; Mention @.>
Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)
per the wiki: https://github.com/TheJacksonLaboratory/cs-nf-pipelines/wiki/GBRS-Pipeline-ReadMe
/gbrs/.interpolated.genoprobs.tsv: Interpolated genotype probabilities in tab delimited format
Beyond this, I am not able to help with your analysis.
—
Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2271533665, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MS24E4GZPDDQRZHJW43ZQDRSLAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZRGUZTGNRWGU.
You are receiving this because you were mentioned.


support cabal new-build

Built on top of #8.

This looks sensible to me, although I'm particularly strong with bash nor travis. It looks like travis did the right thing.
I've added a comment about running the tests with new-test.

Thanks a lot for the review, @gwils! I've merged this, could you somehow test that this works for you now from master? If yes, I'll cut a release to hackage.

Yes, master successfully builds and passes tests for me with new-build. Thanks!

@gwils: Released as <IP_ADDRESS>. Thanks for your help.


accept pagination args in #getTxHistory

Coverage increased (+0.08%) to 95.59% when pulling 20953d94c9ae6c5d3bffeba9d0e9192d80651e52 on isocolsky:feat/history_pagination into 7b56eb51f9c7da1082a5198d12a711d6f76ef6e8 on bitpay:master.


Chore: Separate repl into separate project, fix REPL docs output

This PR separates the REPL into its own sub-library which helpful for fixing several issues:

This improves the compile times of pact-5 for the sake of chainweb integration.
It allows us to pull in pandoc, which fixes in our docs issues where our docs in the repl look like the raw markdown docs.

Before this PR:

After this PR:

PR checklist:

[x] Test coverage for the proposed changes
[x] PR description contains example output from repl interaction or a snippet from unit test output
[x] (If Relevant) Documentation has been (manually) updated at https://docs.kadena.io/pact

Additionally, please justify why you should or should not do the following:

[x] Benchmark regressions

N/A.

[x] Confirm replay/back compat (Ignore until core release)
[x] (For Kadena engineers) Run integration-tests against a Chainweb built with this version of Pact (Ignore until core release)

As of my last commit, it also fixes an annoying issue in the pact-5 repl that repl functions were actually going through the gas charging method, which when it actually charges gas, will essentially brick the repl.
The following is now possible:
pact>(env-gasmodel "table")
"Set gas model to table-based cost model"
pact>(env-gaslimit 1)
"Set gas limit to 1"
pact>(env-gas
env-gas             env-gaslimit        env-gaslog          env-gasmodel        env-gasmodel-fixed
pact>(+ "hello" "world")
(interactive):1:0: Gas limit "1"exceeded: 2
 1 | (+ "hello" "world")
   | ^^^^^^^^^^^^^^^^^^^

pact>(env-gas 0)
"Set gas to 0"


Error: read ECONNRESET

Hi guys, after sometime my worker is showing this error:
error: socket error: Error: read ECONNRESET
    at exports._errnoException (util.js:870:11)
    at TCP.onread (net.js:552:26)

Any idea?

Maybe your server is down.

I don't know why but this only occurs on Windows, Linux stay stable always.

please write a deterministic scenario/test how to simulate it, there is not enough informations in your post

I don't use Gearmanode on windows anymore. On ubuntu server it's working fine.

ok, thx for info


Bugfix/#136

Fixed several issues when it comes to words
இருபத்திரண்டு
முப்பத்து மூன்று
நாற்பத்து நான்கு
ஐம்பத்தைந்து
அறுபத்தாறு
etc., The only problem I see is, because of the way the spaces are not cleanly added, it is getting messed up in some cases, the Django webpage renders them properly but python code see extra spaces in very limited places. I have updated the Unit test cases for that. The spacing needs a separate cleanup work to make it consistent.

Hi Sathia ... I'm not able to integrate if unit tests are not passing. Please update the unit tests and make them pass. I'll take a more closer look after that.


GH1213: Improve NuGetPushSettings.Source documentation

Document the fact that NuGetPushSettings.Source is mandatory since nuget 3.4.2 and also document the url for two of the most popular public nuget servers (nuget.org and myget).
Resolves #1213

Done. Encoded the < and > in the XML comments

@Jericho  your changes have been merged, thanks for your contribution 👍


Lookup: Enable automatic space to '-' conversion upon note creation.

Please select if your request is either something new or an enhancement

[x] Enhancement of an existing Feature.
[ ] Request of a new feature.

Please select the area your request applies to. (Multiple selections are Possible. You can leave blank if you're not sure.)

[ ] Workspace - VSCode workspace, vaults, Intellisense/autocomplete, Dendron settings
[x] Lookup - Dendron's Lookup Command
[ ] Views - Dendron Preview, Tree View, Side Panels in the UI
[ ] Schema - Dendron Schemas
[ ] Pod - Data import from / export to Dendron
[ ] Publish - External Site Publish
[ ] Markdown - Markdown features, such as syntax support and features

Is your feature request related to a problem? Please describe
Dendron naming philosophy suggesting using '-' and lowercase letters. Typing '-' is way less convenient than pressing space. Hence why not enable configuration that will automatically replace space with '-' to make note creation more convenient.
Describe the solution you'd like
Add new lookup configuration which when set to 'true' will replace space with '-' in created file names. The 'title' of the note should stay the same. This also would be very useful when pasting some title from external documentation to create a note.
Describe alternatives you've considered
Typing things out manually.

Keep the crop suggestions coming!


⬆ Upgrades python3 to 3.10.5-r0

Proposed Changes
SSIA
Related Issues

(Github link to related issues or pull requests)

What happened with the indentations? 😄

Probably vscode reformats the file, let me fix it


UniqueConstraints and Django 5

Greetings, I just found out a bug when using UniqueConstraints.
Let's say you have a model, when you want 2 fields to be unique together (either way works).
The modern one:
class MyModel(models.Model):
    class Meta:
        constraints = [
            models.UniqueConstraint(fields=["field1", "field2"], name="unique_mymodel")
        ]

    field1 = models.CharField(max_length=255)
    field2 = models.CharField(max_length=255)
    objects = CopyManager()

The could be deprecated one:
class MyModel(models.Model):
    class Meta:
        unique_together = ("field1", "field2")

    field1 = models.CharField(max_length=255)
    field2 = models.CharField(max_length=255)
    objects = CopyManager()

And then you try to use this awesome library to load some big stuff, but you get this error:
MyModel.objects.from_csv("my_cool_csv.csv")

File "myfile.py", line 163, in my_function
  MyModel.objects.from_csv("my_cool_csv.csv")
File "my-project/env/lib/python3.10/site-packages/django/db/models/manager.py", line 87, in manager_method
  return getattr(self.get_queryset(), name)(*args, **kwargs)
File "my-project/env/lib/python3.10/site-packages/postgres_copy/managers.py", line 182, in from_csv
  self.drop_indexes()
File "my-project/env/lib/python3.10/site-packages/postgres_copy/managers.py", line 81, in drop_indexes
  f"Dropping index_together of {self.model._meta.index_together}"
AttributeError: 'Options' object has no attribute 'index_together'. Did you mean: 'unique_together'?

It works fine for LTS django 4.2, it fails on the latest one 5.1.1. For now because I can't survive without your lib I switched back to the LTS version.
I think an if clause checking for existing attributes could fix the issue, not sure though.

Good catch. It looks like index_together has been removed from Django 5.1, and looks like unique_together is depreciated but still supported.

Hmm. Should we drop that bit from our code here?

@palewire I can submit a PR shortly that uses getattr for both index_together and unique_together if you'd like.

That would be great.

This should be fixed in the latest release.


The Composer installer script did not run correctly [exit code 259] and no output was returned.

The Composer installer script did not run correctly [exit code 259] and no output was returned.

Unless you supply the required information (the log file) there is not much we can so.


Color struct types with measure as value types

This fixes https://github.com/Microsoft/visualfsharp/issues/3110
It colors any type having a measure attached as value type. I've tried to determine is the TyconRef at

is a value type, but failed. The entity (for example, int'1 for int<M>) is not a structure and it's not an abbreviation, so I was stuck.

Solved:

Even Roslyn tests are not launched on CI?... wow

@dsyme Thanks you. Done.

@vasily-kirichenko I modified the code to this:
    let isStructTyconRef (tyconRef: TyconRef) =
            let ty = generalizedTyconRef tyconRef
            let underlyingTy = stripTyEqnsAndMeasureEqns g ty
            isStructTy g underlyingTy

Oddly isStructTy is returning false on things like int32<kg>, I need to check separately if that is causing problems anywhere.  On a quick look for most of the places where it might matter we have explicit calls to stripTyEqnsAndMeasureEqns

@dsyme I've checked, your change works ok.

@dotnet-bot test Ubuntu16.04 Release_default Build please


pymongoarrow does not return nested fields

I'm trying out pymongoarrow to fetch a large dataset, with regular pymongo, I'm able to get all the fields that I needed, but with pymongoarrow it seems to not return any field with nested values (e.g. dict), did I miss something?

Thanks @samshipengs, support for nested fields was merged recently (in https://github.com/mongodb-labs/mongo-arrow/commit/b9c5164037a4e6280e17597f8e5ca6ddf137c6d1) and will be released soon.

Good to hear that, thanks!


Sheet: selectItems of SheetColumn evaluation on cell instead of column level

XHTML:
<pe:sheet
          id="sheet"
          widgetVar="sheetWidgetVar"
          value="#{bean.persons}"
          var="person"
          rowKey="#{person.id}"
          showRowHeaders="true"
          styleClass="sheet">
    <pe:sheetcolumn
                    headerText="instrument"
                    value="#{person.instrument}"
                    colType="autocomplete"
                    selectItems="#{instrumentList.getEnabledInstrumentsIncluding(person)}"
                    autoCompleteStrict="false"
                    autoCompleteVisibleRows="10" />
</pe:sheet>

public class Person {

private String name;

private Instrument instrument;
}

public class Instrument {

private String name;

private boolean enabled;
}

public class InstrumentList {

public List<Instrument> getEnabledInstrumentsIncluding(Person person) {
    return (List<Instrument>) service.getEnabledInstrumentsIncluding(person).getResultList();
}

Use case:
In an excel sheet, only enabled values should be shown in the autocomplete. However, if in the future an instrument is/will be disabled, the autocomplete does not show said instrument anymore in the autocomplete, yet the current instrument still represents a valid selection. Therefore the method gets all the enabled instruments including the one already on the person object.
Current behavior:
The method 'getEnabledInstrumentsIncluding(person)' is called/evaluated on column level, i.e., it is only called once and hence person always resolves to null. This makes it impossible to populate an autocomplete on the individual cell level.
Expected behavior:
The method 'getEnabledInstrumentsIncluding(person)' is called/evaluated on cell level.

I don't think the way it is currently designed it can be done at the cell level.  I can take a look but I think this was done for performance because the way the sheet works all the JSON data is sent to the sheet up front and not per cell as you click on it.

@melloware
Thanks for the information. This was my initial thought. Thanks for looking into it.


App not asking for file permission which allows it to download gallery's

Your issue may already be reported!
Please search on the issue tracker before creating one.
Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

Version used:
Operating System version:

NHentai link (if relevant):

Same issue


JsonString in schema vs in json

In schema:
"Permission": {
  "Struct": [
    {
      "name": "name",
      "type": "String"
    },
    {
      "name": "payload",
      "type": "JsonString"
    }
  ]
},

but in genesis.json:
{
  "Grant": {
    "Permission": {
      "object": {
        "name": "CanSetParameters",
        "payload": null
      },
      "destination": "ed0120CE7FA46C9DCE7EA4B125E2E36BDB63EA33073E7590AC92816AE1E861B7048B03@wonderland"
    }
  }
},

Expected result
Schema:
"Permission": {
  "Struct": [
    {
      "name": "name",
      "type": "String"
    },
    {
      "name": "payload",
      "type": "Option<JsonString>"
    }
  ]
},

Logs

Log contents
Replace this text with a JSON log,
so it doesn't grow too large and has highlighting.

Who can help to reproduce?
@mversic
Notes
No response

I'm not sure if this is a bug. The problem here is that schema is a description of how to serialize data model as SCALE.  When serializing JsonString to SCALE we serialize it as a string. However, when serializing to json the format is completely different. I'm not sure how we can reconcile this. JsonString is serialzed as a json node so marking the field as Option<JsonString> is not the solution in this case. Consider this:
{
    "name": "CanRemoveKeyValueInAccount",
    "payload": {
        "account": "ed0120CE7FA46C9DCE7EA4B125E2E36BDB63EA33073E7590AC92816AE1E861B7048B03@wonderland"
    }
}

what is type of payload here? it's not string or null, it's a map. And what about this case:
{
    "name": "CanRemoveKeyValueInAccounts",
    "payload": ["ed0120CE7FA46C9DCE7EA4B125E2E36BDB63EA33073E7590AC92816AE1E861B7048B03@wonderland"]
}

the type is now array

In JSON, I think it is perfectly fine to have any JSON value for fields that are JsonString, including null.
The problem again is that the schema does really tell only how to implement SCALE encoding. And I think it's fine to keep the schema as is.
@arndey, what problem does it create for you specifically?

@0x009922  that's the first case when value of the field can be null but the type in the schema not optional, so the problem is because we generate our models from your schema and we can't operate nullable payload

Why it can be nullable here, but can't in Permission?
/**
 * ExecuteTriggerEvent
 *
 * Generated from 'ExecuteTriggerEvent' regular structure
 */
public data class ExecuteTriggerEvent(
    public val triggerId: TriggerId,
    public val authority: AccountId,
    public val args: String? = null,
)

"ExecuteTriggerEvent": {
    "Struct": [
      {
        "name": "trigger_id",
        "type": "TriggerId"
      },
      {
        "name": "authority",
        "type": "AccountId"
      },
      {
        "name": "args",
        "type": "Option<JsonString>"
      }
    ]
  }

IMO Alternatively, for consistency Iroha should replace Option<JsonString> with just JsonString so that it's a nullable everywhere. Current contradiction is that you treat JsonString generally nullable but wrap it with Option somewhere.

IMO Alternatively, for consistency Iroha should replace Option<JsonString> with just JsonString so that it's a nullable everywhere. Current contradiction is that you treat JsonString generally nullable but wrap it with Option somewhere introducing a false guess it should be treated nullable only if it's wrapped.

this will solve the current problem, but in terms of using the SDK, if all the fields are nullable, it will not be very convenient, it seems to me

this will solve the current problem, but in terms of using the SDK, if all the fields are nullable, it will not be very convenient, it seems to me

you mean any field nullability should be highlighted via wrapping with Option?

this will solve the current problem, but in terms of using the SDK, if all the fields are nullable, it will not be very convenient, it seems to me

you mean any field nullability should be highlighted via wrapping with Option?

yes, and so far I don’t understand what the problem is, given the examples that I wrote earlier here

The source of truth is the data model. In it, we decided to encode JSON values as plain strings. Some structures require some payload to be (like permissions) and it is expressed as JsonString in the schema; others have it optionally (like trigger events) - it is expressed as Option<JsonString>.
Strictly speaking, JsonString with value "null" is not the same as Option<String> with value None or Some("null").
Why not make Option<JsonString> everywhere? Iroha can have a special-case handling, treating Option<JsonString> of value None the same as Some("null"), right?
Yes, it can. However, from the standpoint of the data model - why?.. This need doesn't come from the data model, which is the source of decisions. Moreover, this adds an implicit convention to the schema instead of relying on explicit behaviour "baked" into it by design.
The problem we face is that we actually added an implicit convention to how JsonString is represented in JSON. If we were strict, it would be like this:
{
  "payload1": "null", // `JsonString` => `"null"`
  "payload2": null,   // `Option<JsonString>` => `None`
  "payload3": "null", // `Option<JsonString>` => `Some("null")`,
  "payload4": "{\"foo\": \"bar\"}" // not really nice :/
}

But we (de)serialize JsonString as just JSON value in JSON format, and here is the ambiguity:
{
  "payload1": null,   // `JsonString` => `"null"`
  "payload2": null,   // `Option<JsonString>` => `None`
                      //  or should it be `Some("null")`?
  "payload3": "null", // `Option<JsonString>` => `Some("\"null\"")`,
  "payload4": { "foo": "bar" } // but this one is more convenient!
}

While it is convenient for most cases, it is impossible to distinguish None vs Some("null") - both are nulls  in JSON repr.
Solutions?

Make Option<JsonString> everywhere - meh :/
Treat JsonString as a string in JSON

No ambiguity
Less convenient

Change convention: add special representation for Option<JsonString> in JSON:

None - null
Some(value) - { "json": value }

{
  // `None`
  "payload1": null,
  // `Some("null")`
  "payload2": { "json": null },
  // possible to set via plain json values
  "payload3": { "json": { "foo": "bar" } },
  "payload4": "null", //  DISALLOWED,
}

Thanks for the exhaustive overview @0x009922.
Solution 3 seems to be the most convenient from the perspective of how serialized objects look.
What about replacing Option<JsonString> with JsonNode (JsonObject)? so it's either null or {..}, not sure about Arrays though and whether we need them for the top level.

What about replacing Option<JsonString> with JsonNode (JsonObject)? so it's either null (None) or {..} (Some(obj)), not sure about Arrays though and whether we need them for the top level.

@Mingela you mean to replace Option<JsonString> with JsonObject in the schema, treating null and objects differently? I don't see how it's going to solve the same ambiguity when someones needs to set a null payload (not an absent payload). Moreover, this solution kind of doesn't belong to the data model design itself, which I explained above, if that makes sense.
In order to strictly distinguish payload absense vs null payload (it is not the same!) while preserving inlining payloads as JSON object and not as a string, the solution is to use null vs { someKey: payload }, or null vs [payload], or any other value vs a container with actual payload which are strictly distinguishable by shape.

I don't see how it's going to solve the same ambiguity when someones needs to set a null payload

Here we should just decide about a nullable type to be used. I guess the challenge is introduced because the schema is oriented to Rust type system explicitly which has been discussed (and postponed) many times already.

payload absense vs null payload (it is not the same!)

I believe that statement might introduce a confusion. As for the schema definition of course if such a field is defined it should be expected by 'the schema implementing type system'. Though, any not mandatory attribute can be initialized to null by default even if not present in the deserializable object at all.
Going back to the initial problem of ambiguity of Option<JsonString> vs JsonString (null vs "null") the latter value "null" should be indeed prohibited. What is the benefit of treating None vs Some("null") differently? I can't get that really. It seems a Rust implementation implication that should be refactored/optimized.
So, according to the solution 3 suggested, any JsonString value would contain an object ({..} or [{..},..,{..}], not "null") and for the attributes where nulls are possible it should be Option<JsonString>.

@mversic I couldn't see how your examples are relevant though, by the current schema the values should be:

"payload": "{\"account\": \"ed0120CE7FA46C9DCE7EA4B125E2E36BDB63EA33073E7590AC92816AE1E861B7048B03@wonderland\"}"
"payload": "[\"ed0120CE7FA46C9DCE7EA4B125E2E36BDB63EA33073E7590AC92816AE1E861B7048B03@wonderland\"]"
but Andrey refers to null value shown in the genesis, not "null"

@mversic I couldn't see how your examples are relevant though, by the current schema the values expected are:

I need to correct you, you have put extra quotes. JsonString is not serialized as a string when serialized to JSON but rather as a json value node. It is serialized as string when encoded to SCALE. This is the jist of the problem. Schema is a description of how to represent data in SCALE, not json

* `"payload": {\"account\":\"ed0120CE7FA46C9DCE7EA4B125E2E36BDB63EA33073E7590AC92816AE1E861B7048B03@wonderland\"}`
* `"payload": [\"ed0120CE7FA46C9DCE7EA4B125E2E36BDB63EA33073E7590AC92816AE1E861B7048B03@wonderland\"]`

I can understand the concept but there is no indication of how it should be treated by the client given we encounter both Option<JsonString> and JsonString possibly having the same value (null) implying JsonString gets (de)serialized differently (sometimes to JSON, sometimes to SCALE). Shouldn't we rely on distinct types respectively to the usage? Is that already achieved via Option wrapper meaning that's the SCALE and just JsonString is always JSON (probably not)?
@mversic what would be you suggestion on how to overcome this confusion? There have been multiple options mentioned in the discussion. We'd like this issue to get the target solution, preferably with a timeline assigned.

 Please consider splitting JsonString into JsonString + JsonNode or smth then.

yeah, we need to rename it. I suggest JsonString -> JsonObject

@mversic what would be your suggestion on how to overcome this confusion? There have been multiple options mentioned in the discussion. We'd like this issue to get the target solution, preferably with a timeline assigned.

I suggest that JsonString is special handled by the sdk. That is, it's schema is implied by it's name and not read from the file

I suggest that JsonString is special handled by the sdk. That is, it's schema is implied by it's name and not read from the file

Agree on that.
There is no issue with schema in terms of SCALE encoding. JSON encoding according to schema is already a special case handled by Java SDK, so why not handle JsonString specifically too?

I advocate for the following solution:
Minimally: do not inline JsonString in JSON as an actual JSON node. Just leave at as a string with valid JSON inside. This removes the ambiguity and does not require any handling from SDK side.
// JsonString
"null"
"{\"foo\":false}"
"[1, 2, 3]"

// Option<JsonString>
null
"null" // note that it is different from the above
"[1, 2, 3]"

Preserving inline mode: on top of the minimal solution, allow a special notation in JSON in a form of { __inline__: ... }. This will require a special handling on SDK side - parse JsonString either as a string or extract a JSON node from the recognisable shape.
// JsonString
"null"
{ __inline__: null } // equivalent
"[1, 2, 3]"
{ __inline__: [1, 2, 3] } // equivalent

// Option<JsonString>
null                 // None
"null"               // Some("null")
{ __inline__: null } // Some("null")

Recommendation for SDK: handle JsonString in a special way - not as a plain string, but as a custom class implementing platform-native JSON handling. Example from iroha-javascript:
https://github.com/0x009922/iroha-javascript/blob/a895a17a0758a8802d68998a6f3c54689ba15b7f/packages/data-model/src/datamodel/core.ts#L164-L217
Note this specifically - it's just a string in SCALE, but then wrapped into a convenient class on top of it:
export const Json$codec: Codec<Json> = String$codec.wrap(
  (json) => json.asJsonString(),
  (str) => Json.fromJsonString(str),
)

Optional: rename JsonString to JsonValue in the schema. I am not sure about this though - effectively, it is still just a string in terms of SCALE encoding. The schema's only purpose is to describe SCALE encoding.